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RTO  is  the  single  focus  in  NATO  for  Defence  Research  and  Technology  activities.  Its  mission  is  to  conduct  and  promote 
co-operative  research  and  information  exchange.  The  objective  is  to  support  the  development  and  effective  use  of 
national  defence  research  and  technology  and  to  meet  the  military  needs  of  the  Alliance,  to  maintain  a  technological 
lead,  and  to  provide  advice  to  NATO  and  national  decision  makers.  The  RTO  performs  its  mission  with  the  support  of  an 
extensive  network  of  national  experts.  It  also  ensures  effective  co-ordination  with  other  NATO  bodies  involved  in  R&T 
activities. 


RTO  reports  both  to  the  Military  Committee  of  NATO  and  to  the  Conference  of  National  Armament  Directors. 
It  comprises  a  Research  and  Technology  Board  (RTB)  as  the  highest  level  of  national  representation  and  the  Research 
and  Technology  Agency  (RTA),  a  dedicated  staff  with  its  headquarters  in  Neuilly,  near  Paris,  France.  In  order  to 
facilitate  contacts  with  the  military  users  and  other  NATO  activities,  a  small  part  of  the  RTA  staff  is  located  in  NATO 
Headquarters  in  Brussels.  The  Brussels  staff  also  co-ordinates  RTO’s  co-operation  with  nations  in  Middle  and  Eastern 
Europe,  to  which  RTO  attaches  particular  importance  especially  as  working  together  in  the  field  of  research  is  one  of  the 
more  promising  areas  of  co-operation. 


The  total  spectrum  of  R&T  activities  is  covered  by  the  following  7 
•  AVT  Applied  Vehicle  Technology  Panel 
Human  Factors  and  Medicine  Panel 
Information  Systems  Technology  Panel 
NMSG  NATO  Modelling  and  Simulation  Group 
SAS  System  Analysis  and  Studies  Panel 

Systems  Concepts  and  Integration  Panel 
Sensors  and  Electronics  Technology  Panel 
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bodies: 


These  bodies  are  made  up  of  national  representatives  as  well  as  generally  recognised  ‘world  class’  scientists.  They  also 
provide  a  communication  link  to  military  users  and  other  NATO  bodies.  RTO’s  scientific  and  technological  work  is 
carried  out  by  Technical  Teams,  created  for  specific  activities  and  with  a  specific  duration.  Such  Technical  Teams  can 
organise  workshops,  symposia,  field  trials,  lecture  series  and  training  courses.  An  important  function  of  these  Technical 
Teams  is  to  ensure  the  continuity  of  the  expert  networks. 


RTO  builds  upon  earlier  co-operation  in  defence  research  and  technology  as  set-up  under  the  Advisory  Group  for 
Aerospace  Research  and  Development  (AGARD)  and  the  Defence  Research  Group  (DRG).  AGARD  and  the  DRG  share 
common  roots  in  that  they  were  both  established  at  the  initiative  of  Dr  Theodore  von  Karman,  a  leading  aerospace 
scientist,  who  early  on  recognised  the  importance  of  scientific  support  for  the  Allied  Armed  Forces.  RTO  is  capitalising 
on  these  common  roots  in  order  to  provide  the  Alliance  and  the  NATO  nations  with  a  strong  scientific  and  technological 
basis  that  will  guarantee  a  solid  base  for  the  future. 
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Executive  Summary 

When  MSG-0 19/TG-0 16  was  established,  both  the  NATO  and  the  distributed  simulation  communities  had 
realized  a  need  for  guidance  about  the  Verification,  Validation,  and  Accreditation  (VV&A)  of  federations. 
The  efforts  of  this  Task  Group  addressed  this  need  for  both  communities  by  developing  a  baseline  view  of 
VV&A  which  emphasized  the  lack  of  a  universal  model  for  federation  VV&A  processes  due  to  divergent 
national  perspectives  and  needs.  The  Task  Group  determined  that  the  lack  of  consistent  methodology  was 
hampering  NATO’s  ability  to  develop  shared  simulation  federations.  Resolving  these  disparities  required 
developing  a  consistent  understanding  of  what  VV&A  is  and  how  to  apply  it  to  achieve  the  best  results. 
Despite  member  nation’s  different  perspectives,  they  were  able  to  agree  on  a  single  unified  model  of 
federation  VV&A  processes.  Thus,  the  baseline  and  the  results  from  the  Combined  Convention  on 
International  VV&A  Standardization  Endeavours  (CConVV&A)  resulted  in  a  decision  to  merge  the 
efforts  of  this  Task  Group  and  the  Simulation  Interoperability  Standards  Organization  (SISO)  VV&A 
Overlay  drafting  group.  This  merge  expanded  the  focus  of  this  Task  Group  beyond  preparing  a  technical 
report  to  actually  producing  a  VV&A  Overlay  standard  product.  The  Task  Group  also  identified  several 
terms  needed  to  describe  VV&A  processes  for  federations  and  developed  definitions  for  these  terms  using 
existing  standard  definitions. 

Formalizing  the  Overlay  as  an  Institute  of  Electrical  and  Electronics  Engineers  (IEEE)  standard  was 
determined  to  be  the  most  appropriate  standardization  approach  as  it  provides  international  recognition  and 
access;  co-locates  the  Overlay  with  its  companion  document  IEEE  standard  1516.3  for  the  High  Level 
Architecture  (HLA)  Federation  Development  and  Execution  Process  (FEDEP);  and  provides  maintenance 
and  management  of  the  Overlay  by  IEEE.  The  proposed  IEEE  standard  (1516.4)  addresses  the  needs  of  both 
the  NATO  and  SISO  communities  for  a  W&A  Overlay  to  the  FEDEP.  This  Task  Group  succeeded  through 
extensive  collaboration  with  other  international  groups  including  SISO,  Western  European  Armament  Group 
Research  and  Technology  program  on  VV&A  (REWA)  and  International  Test  Operations  Procedures 
(ITOP).  The  tight  coupling  between  the  NATO  and  SISO  groups  resulted  in  a  better,  more  widely  applicable 
Overlay  (e.g.,  government,  industry,  and  academia).  The  collaboration  benefited  NATO  by  directly  coupling 
the  Modelling  &  Simulation  Group  (MSG)  product  to  an  internationally  recognized  standards  body  and 
benefited  SISO  by  ensuring  early  and  extensive  international  input  to  the  final  product.  This  collaborative 
effort  could  serve  as  a  model  for  future  joint  Modelling  &  Simulation  (M&S)  standards  initiatives. 

While  the  Task  Group  feels  that  enormous  strides  have  been  made  with  the  development  of  the  VV&A 
Overlay,  still  more  work  is  needed  to  increase  the  effectiveness  and  efficiency  of  the  VV&A  processes. 
Particular  areas  to  be  addressed  include  case  studies  to  mature  the  Overlay,  tailoring  the  VV&A  processes 
based  on  user  risk  management,  and  effective  communication  of  risk  to  program  management. 

Based  on  the  above  conclusions,  the  Task  Group  makes  the  following  recommendations: 

1)  NATO  should  adopt  Appendix  1  as  an  interim  Standardization  Agreement  (STANAG)  until  the 
IEEE  1516.4  standard  is  issued  and  then  revise  the  STANAG  to  adopt  the  IEEE  standard. 

2)  MSG-054,  the  follow-on  study  group,  should  participate  in  the  IEEE  1516.4  balloting  process, 
monitor  on-going  VV&A  case  study  initiative  and  incorporate  suggested  improvements  to  the 
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Overlay  during  the  balloting  process,  and  study  the  relationship  between  user  risk,  acceptance 
criteria,  and  tailoring  of  the  VV&A  process. 

3)  NATO  should  consider  adopting  similar  collaborative  relationships  with  international  standards 
bodies  as  appropriate. 
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Synthese 

Lorsque  le  MSG-019/TG-016  fut  mis  en  place,  l’OTAN  comme  les  communautes  de  simulation  distribuee 
se  sont  rendues  compte  de  la  necessite  d’un  guide  pour  la  Verification,  la  Validation  et  TAccreditation 
(VV&A)  des  federations.  Les  efforts  de  ce  groupe  de  recherche  ont  fait  part  de  ce  besoin  aux  deux 
communautes  en  developpant  une  vision  de  reference  sur  la  VV&A  qui  mettait  en  relief  le  manque  de 
modele  universel  pour  la  Verification,  la  Validation  et  TAccreditation  des  federations  a  cause  des 
perspectives  et  besoins  nationaux  divergents.  Le  groupe  de  recherche  a  determine  que  le  manque  de 
methodologie  coherente  diminuait  la  faculte  de  l’OTAN  a  developper  des  federations  partagees  de 
simulation.  Resoudre  ces  disparites  necessitait  de  developper  une  comprehension  coherente  de  ce  qu’est  la 
VV&A  et  comment  l’appliquer  pour  obtenir  les  meilleurs  resultats.  En  depit  des  perspectives  differentes 
des  nations  membres,  elles  purent  s’accorder  sur  un  modele  unifie  a  propos  de  la  VV&A  des  federations. 
Ainsi,  la  reference  et  les  resultats  de  la  convention  combinee  sur  les  efforts  de  normalisation  de  la  VV&A 
intemationale  (CConVV&A)  aboutirent  a  une  decision  de  fusionner  les  efforts  de  ce  groupe  de  recherche 
et  du  groupe  preparatoire  de  l’organisation  des  normes  d’interoperabilite  de  la  simulation  (SISO)  VV&A 
Overlay.  Cette  fusion  etendit  le  champ  de  ce  groupe  de  travail  au-dela  de  la  preparation  d’un  rapport 
technique  pour  effectivement  produire  une  norme  VV&A  Overlay.  Ce  groupe  de  travail  a  aussi  identifie 
plusieurs  termes  necessaires  pour  definir  la  VV&A  pour  les  federations  et  a  mis  au  point  une  definition  de 
ces  termes  en  utilisant  des  definitions  standard  existantes. 

On  a  determine  que  formaliser  cette  Overlay  sous  la  forme  d’une  norme  IEEE  (institut  des  ingenieurs 
electriciens  et  electroniciens)  etait  l’approche  normative  la  plus  appropriee,  car  cela  permet  de  disposer 
d’une  reconnaissance  et  d’un  acces  international ;  cela  permet  de  rapprocher  l’Overlay  des  documents 
voisins  IEEE,  a  savoir  :  Standard  1516.3  for  the  High  Level  Architecture  (HLA)  Federation  Development 
and  Execution  Process  (FEDEP)  ;  cela  permet  aussi  d’entretenir  et  de  gerer  l’Overlay  selon  la  norme 
IEEE.  La  norme  proposee  IEEE  (1546.4)  resout  a  la  fois  les  besoins  de  l’OTAN  et  ceux  des  communautes 
SISO  pour  1’ Overlay  VV&A  par  rapport  au  FEDEP.  Ce  groupe  de  recherche  a  reussi  grace  a  une 
collaboration  extensive  avec  d’autres  groupes  intemationaux  comprenant  SISO,  Western  European 
Armament  Group  Research  et  Technology  program  on  VV&A  (REVVA)  et  International  Test  Operations 
Procedures  (ITOP).  Cette  etroite  cooperation  entre  les  groupes  OTAN  et  SISO  a  abouti  a  une  meilleure 
Overlay  (administration,  industrie  et  universites)  plus  generalement  applicable.  Cette  collaboration  a 
profite  a  l’OTAN  en  couplant  directement  le  produit  MSG  (groupe  de  modelisation  &  simulation)  a  un 
organisme  normatif  reconnu  intemationalement,  et  au  SISO  en  lui  dormant  un  acces  precoce  et  international 
au  produit  final.  Cet  effort  de  collaboration  pourrait  servir  de  modele  a  des  initiatives  interarmees  de 
normalisation  en  matiere  de  modelisation  et  de  simulation  (M&S). 

Tandis  que  le  groupe  de  recherche  a  le  sentiment  que  des  pas  de  geant  ont  ete  faits  pour  developper  la 
VV&A  Overlay,  il  n’en  demeure  pas  moins  que  du  travail  supplementaire  est  necessaire  pour  augmenter 
Tefficacite  des  processus  VV&A.  Les  domaines  particuliers  dont  il  faut  s’occuper  incluent  des  etudes  de 
cas  pour  faire  murir  TOverlay  en  « taillant  sur  mesure  »  les  processus  VV&A  d’apres  la  gestion  des 
risques  par  l’utilisateur  et  la  communication  effective  du  risque  sur  la  gestion  du  programme. 
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Compte  tenu  des  conclusions  ci-dessus,  le  groupe  de  recherche  fait  les  recommandations  suivantes  : 

1)  L’OTAN  devrait  adopter  l’Appendice  1  comme  accord  temporaire  de  normalisation  (STANAG) 
jusqu’a  ce  que  la  norme  IEEE  1516.4  soit  publiee,  puis  reviser  le  STANAG  pour  adopter  la  norme 
IEEE. 

2)  Le  MSG-054,  groupe  de  suivi  d’etude,  devrait  participer  au  vote  sur  le  1516.4  de  1’IEEE,  surveiller 
1’ initiative  d’etude  du  cas  VV&A  et  incorporer  les  ameliorations  suggerees  a  1’ Overlay  lors  du  vote, 
puis  etudier  les  relations  entre  les  risques  utilisateur,  les  criteres  de  reception  et  Tadaptation 
«  sur-mesure  »  de  la  VV&A. 

3)  L’OTAN  devrait,  au  besoin,  envisager  de  collaborer  de  la  meme  fag  on  avec  les  organismes 
normatifs  intemationaux. 
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ORGANIZATION 


1.1  BACKGROUND 

The  idea  for  an  Exploratory  Team  on  Verification,  Validation,  and  Accreditation  (VV&A)  of  Federations  was 
first  proposed  during  the  5th  NATO  Modelling  and  Simulation  Group  (NMSG)  meeting  in  July  2000. 
Following  initial  preliminary  discussions  and  a  short  meeting  by  interested  parties  in  September  2000  at  the 
United  States  Defense  Modeling  and  Simulation  Office  (DMSO),  it  became  clear  that  sufficient  interest  and 
substance  existed  for  the  formation  of  a  Task  Group  (TG)  on  the  subject. 

Credibility  is  critical  to  the  effective  and  appropriate  use  of  modelling  and  simulations  (M&S)  when  used 
independently  or  combined  in  a  federation.  The  cornerstone  to  establishing  M&S  credibility  is  a  robust 
VV&A  process.  There  is  a  widely  disparate  understanding  of  the  VV&A  process  within  the  NATO/Partners 
for  Peace  (PfP)  community.  Therefore  the  need  exists  to  establish  a  consistent  baseline  of  terminology, 
philosophy,  and  methodology. 

Building  a  federation  that  incorporates  representations  appropriate  to  the  needs  of  its  users  relies  heavily  on 
the  information  generated  by  the  VV&A  process.  Just  as  the  High  Fevel  Architecture  (HFA)  [1]  provides  the 
framework  that  addresses  technical  interoperability  (e.g.,  issues  related  to  connectivity  and  data  exchange), 
a  well-defined  VV&A  process  supports  the  establishment  of  substantive  interoperability  (e.g.,  issues  related  to 
representations,  consistency). 

The  NATO/PfP  community  currently  has  several  major  initiatives  that  require  the  development  of  M&S 
federations.  The  success  of  these  initiatives  is  tied  to  the  credibility  of  the  federation  results  and  therefore, 
the  VV&A  process.  This  makes  definition  of  a  consistent  VV&A  process  even  more  imperative. 


1.2  TASKING 

The  charter  of  this  Task  Group  included  seven  distinct  tasks: 

1)  Assess  existing  NATO  and  national  VV&A  products  to  determine  applicability  and  leverage 
potential; 

2)  Determine  VV&A  issues  of  concern  to  the  success  of  major  NATO/PfP  federation  initiatives; 

3)  Formulate  an  outline  describing  the  major  elements  of  the  VV&A  baseline; 

4)  Evolve  the  baseline  into  a  Technical  Report; 

5)  Capture  lessons  learned  from  major  NATO/PfP  initiatives; 

6)  Revise  the  baseline  to  reflect  lessons  learned;  and 

7)  Finalize  the  Technical  Report. 

In  the  course  of  executing  the  first  task,  the  Task  Group  identified,  assessed  and  compared  the  following 
products  and  approaches  related  to  federation  VV&A  or  with  possible  influence  upon  VV&A: 

•  International  Test  Operations  Procedures  VV&A  Documentation  Guidance  [2]; 

•  U.S.  Department  of  Defense  (DoD)  VV&A  Recommended  Practices  Guide  [3]; 
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•  VV&A  of  Federations  by  V.  Dobey  and  R.  Lewis  [4]; 

•  Recommended  Practice  for  VV&A  of  Distributed  Interactive  Simulations,  Institute  of  Electrical  and 
Electronics  Engineers  (IEEE)  1278.4  [5]; 

•  German  V-Model  for  Development  of  Computer-Based  Systems  [6,  7]; 

•  Universitat  der  Bundeswehr  Miinchen  (UBM)  Model  of  VV&A  of  Models  and  Simulations  by 
D.  Brade  [8]; 

•  Guidelines  for  VV&A  Techniques,  THALES  JP  1 1.20  [9,  10,  1 1]; 

•  U.K./France  Joint  Study  of  VV&A  of  Models  and  Simulations  [12]; 

•  UK  Process  for  VV&A  for  Synthetic  Environments  [13,  14]; 

•  U.S.  DoD  Instruction  for  W&A  of  Models  and  Simulations,  DoDI  5000.61  [15]; 

•  Verification  and  Validation  of  Software  Systems,  IEEE1012  [16]; 

•  Director  General  (Scrutiny  &  Analysis)  (DG(SRA))  Guidelines  for  the  Verification  and  Validation  of 
Operational  Analysis  Modelling  Capabilities  (Dec  2002)  [17];  and 

•  Naval  Sea  Systems  Command  (NAVSEA)  Dahlgren  Accreditation  Team  (NDAT)  VV&A  Process 
[18,  19], 

The  Task  Group  then  worked  to  combine  these  various  concepts  for  and  approaches  to  VV&A  and  systems 
development  into  an  outline  for  a  baseline  for  federation  VV&A.  This  outline  evolved  into  a  draft  technical 
report  describing  the  baseline  concepts.  The  structure  of  this  report  closely  paralleled  the  High  Level 
Architecture  (HLA)  Federation  Development  and  Execution  Process  (FEDEP),  IEEE  1516.3  [20]. 

Review  of  this  report  by  a  panel  of  VV&A  experts  as  well  as  by  the  authors  themselves  revealed  the 
disparities  between  the  VV&A  approaches  of  the  member  nations.  The  Task  Group  then  decided  that  a 
consistent  approach  to  the  VV&A  of  federations  could  only  be  successfully  developed  if  it  was  founded  upon 
a  unified  model  of  the  underlying  VV&A  processes. 

Shortly  after  that  decision,  the  First  Combined  Convention  on  International  VV&A  Standardization 
Endeavours  (CConVV&A)  [21]  defined  the  relationship  between  ongoing  VV&A  initiatives  and  their 
associated  products.  These  initiatives  included  Western  European  Armament  Group  Research  and  Technology 
program  on  VV&A  (REVVA),  International  Test  Operations  Procedures  (ITOP),  the  Simulation 
Interoperability  Standards  Organization  (SISO)  VV&A  Overlay  Product  Development  Group  (PDG), 
and  NATO  Modelling  &  Simulation  Group  019  (MSG-019).  As  these  groups  exchanged  information  about 
their  ongoing  efforts  and  the  status  of  their  products,  the  considerable  overlap  between  the  SISO  VV&A  PDG 
and  NATO  MSG-019  became  clear.  One  of  the  recommendations  coming  from  this  convention  was  that  these 
two  efforts  be  combined.  In  January  of  2005,  the  two  groups  met  as  the  VV&A  Overlay  Drafting  Group 
whose  product  would  be  a  VV&A  overlay  to  the  FEDEP. 

This  report  documents  the  results  of  the  MSG-019  Task  Group  efforts  to  establish  the  draft  VV&A  Overlay  of 
the  FEDEP.  The  Overlay  has  entered  the  IEEE  balloting  process  and  is  anticipated  to  become  IEEE  1516.4. 
A  final  draft  of  the  Overlay  is  included  as  an  appendix  to  this  report. 
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1.3  TASK  GROUP  MEMBERSHIP 

Nations  participating  in  the  MSG-019  were  Canada,  France,  Germany,  Sweden,  the  United  Kingdom, 
and  the  United  States.  The  United  States  representative  served  as  chair  of  the  Task  Group. 


1.4  SCOPE  OF  EFFORT 

The  area  of  research  this  Task  Group  focused  on  is  the  VV&A  of  federations.  The  Task  Group’s  scope  was  to 
address  the  issues  most  critical  to  the  success  of  the  aforementioned  NATO/PfP  initiatives.  The  scope  of  this 
effort  included: 

•  Identifying  and  defining  the  terminology  needed  to  clearly  describe  the  VV&A  Overlay  processes; 

•  Characterizing  the  differences  between  federate  VV&A  and  federation  VV&A  and  using  those 
differences  to  distinguish  the  responsibilities  of  those  participating  in  federate  and  federation  VV&A; 

•  Understanding  the  theoretical  nature  of  the  underlying  VV&A  processes  while  emphasizing  the 
practical  application  of  that  theory; 

•  Developing  a  unified  model  of  VV&A  processes  that  addresses  the  disparate  concerns  and  needs  of 
the  member  nations; 

•  Building  VV&A  guidance  based  upon  actual  experience  that  captures  that  anecdotal  but  invaluable 
knowledge; 

•  Recognizing  the  unique  role  that  VV&A  plays  in  the  FEDEP  that  contributes  to  ensuring  that 
federations  can  sufficiently  serve  their  intended  uses; 

•  Capturing  the  assumptions  upon  which  the  guidance  in  the  VV&A  Overlay  is  based;  and 

•  Understanding,  as  much  as  possible,  the  nature  and  extent  of  tailoring  of  the  VV&A  processes. 

The  scope  of  this  research  was  also  constrained  by  the  assumption  that  the  VV&A  Overlay  would  apply 
primarily  to  the  development  and  execution  of  FILA  federations  using  the  FEDEP  [20]. 

While  the  VV&A  Overlay  is  focused  on  the  FEDEP,  users,  developers,  and  VV&A  personnel  working  with 
simulations  and  simulation  compositions  not  based  upon  the  FILA  can  also  benefit  from  the  guidance  in  this 
document  since  the  activities  that  this  overlay  describes  can  support  any  type  of  distributed  simulation 
development. 
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The  VV&A  Overlay  has  been  designed  to  apply  across  a  wide  range  of  functional  applications.  The  purpose 
of  this  Overlay  is  to  provide  a  more  detailed  view  of  the  VV&A  processes  implied  by  the  FEDEP.  Currently, 
these  processes  represent  the  best  practices  available  to  the  VV&A  community.  The  VV&A  overlay  is  a 
tailorable  process  and  is  offered  as  guidance  to  all  participants  in  FEDEP  activities.  Like  the  FEDEP, 
the  activities  of  the  VV&A  Overlay  will  usually  be  executed  iteratively,  rather  than  linearly,  with  the 
successive  refinement  of  its  products.  The  exact  nature  of  any  execution  thread  depends  entirely  upon  the 
intended  use,  the  federation  components  and  the  information  needed  to  support  the  VV&A  processes. 

This  Overlay  identifies  and  describes  the  recommended  VV&A  processes  that  should  be  followed  to  assure 
the  acceptability  and  utility  of  federations  for  particular  intended  uses.  In  order  to  adapt  to  the  needs  of 
individual  member  nations,  this  Overlay  recognizes  that  the  VV&A  processes  must  always  support  some  form 
of  acceptance  of  the  federations  for  their  intended  use,  but  that  only  some  of  those  federations  will  be  formally 
accredited.  The  overlay  also  identifies  and  describes  the  information  feeding  and  resulting  from  those 
processes  as  well  as  the  relationships  between  the  FEDEP  and  the  VV&A  processes  and  their  respective 
information  products.  In  addition,  this  overlay  defines  those  terms  uniquely  needed  to  characterize  the  FEDEP 
VV&A  Overlay.  This  overlay  takes  special  care  to  use  and  build  upon  existing  standards,  standard  terms  and 
their  definitions  whenever  possible. 

Finally,  as  a  convenience,  this  Overlay  uses  the  term  “VV&A  Team”  to  refer  to  those  people  directly  involved 
in  executing  the  federation  VV&A  processes.  The  VV&A  Team  is  not  responsible  for  validating  or  verifying 
the  individual  federates  although  they  do  rely  upon  the  products  of  these  activities  and  may  require 
supplemental  verification  and  validation  to  gain  the  information  they  need  for  federation  VV&A.  The  VV&A 
Team  does  however  have  primary  responsibility  for  federation  validation  activities.  The  membership  of  the 
VV&A  Team  depends  upon  the  needs  of  the  activities  being  executed,  the  personnel  resources  available  and 
the  expectations  of  the  federation  User/Sponsor. 

The  VV&A  activities  described  in  the  overlay,  while  being  generally  applicable  to  most  HLA  federations, 
are  intended  to  be  tailored  to  meet  the  needs  of  each  individual  intended  use.  Every  federation  application  is 
unique.  The  extent  to  which  these  VV&A  processes  can  be  performed  for  a  given  federation  application  will 
depend  on  a  number  of  factors,  including  the  quality  of  the  requirements  information  and  the  resources 
allocated  to  the  VV&A  Team.  The  activities  recommended  in  this  Overlay  should  be  used  as  a  starting  point 
for  developing  the  specific  approach  to  federation  VV&A  needed  to  support  the  intended  use. 

Some  factors  to  be  considered  when  tailoring  VV&A  processes  to  best  fit  the  circumstances  of  each  individual 
application  include: 

•  Tailoring  is  driven  by  risk  and  resource  constraints  (e.g.,  available  information,  budget,  skills,  time); 

•  Quantifying  and  minimizing  risk  are  factors  that  drive  tailoring  decisions; 

•  Risk  associated  with  federation  use  (use  risk)  is  a  function  of  the  quality  of  the  federation  V&V 
information; 

•  The  absence  of  certain  information  input  to  the  VV&A  processes  will  limit  the  quality  of  the  content 
of  the  VV&A  products; 

•  Tailoring  decisions  have  an  impact  upon  use  risk  and  the  resources  required,  and  this  impact  should 
be  understood  and  documented; 
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•  The  VV&A  processes  should  leverage  testing  and  data  collection  tasks  and  describe  this  in  the 
Federation  Accreditation  Plan  and  V&V  Plan; 

•  Any  tailoring  should  adhere  to  any  existing  policies,  standards  or  guidelines  that  may  be  relevant  to 
the  intended  use;  and 

•  Tailoring  may  be  done  at  any  time  in  the  federation  VV&A  processes. 
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IEEE  1516.3,  Recommended  Practice  for  High  Level  Architecture  (HLA)  Federation  Development  and 
Execution  Process  (FEDEP),  describes  an  approach  for  developing  M&S  federations  based  on  sound  systems 
engineering  principles  and  serves  as  the  foundation  for  the  VV&A  Overlay.  Figure  3-1  provides  a  high  level 
view  of  the  major  activities  associated  with  the  FEDEP.  A  description  of  these  steps  follows  Figure  3-1. 
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Figure  3-1:  Top-Level  View  of  the  FEDEP. 


Step  1  -  Define  Federations  Objectives.  The  purpose  of  step  1  of  the  FEDEP  is  to  define  and  document  a  set 
of  needs  that  are  to  be  addressed  through  the  development  and  execution  of  an  HLA  federation  and  to 
transform  these  needs  into  a  more  detailed  list  of  federation  objectives. 

Step  2  -  Perform  Conceptual  Analysis.  The  purpose  of  this  step  of  the  FEDEP  is  to  develop  an  appropriate 
representation  of  the  real  world  domain  that  applies  to  the  federation  problem  space  and  to  develop 
the  federation  scenario.  It  is  also  in  this  step  that  federation  objectives  are  transformed  into  a  set  of  highly 
specific  federation  requirements  that  will  be  used  in  federation  design,  development,  testing,  execution, 
and  evaluation. 
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Step  3  -  Design  Federation.  The  purpose  of  this  step  of  the  FEDEP  is  to  produce  the  design  of  the  federation 
that  will  be  implemented  in  step  4.  This  involves  identifying  existing  federates  that  are  suitable  for  reuse, 
creating  new  federates  and  federate  components  (if  required)  allocating  the  required  functionality  to  federates, 
and  developing  a  detailed  plan  for  federation  development  and  implementation. 

Step  4  -  Develop  Federation.  The  purpose  of  this  step  of  the  FEDEP  is  to  develop  the  Federation  Object 
Model  (FOM),  modify  federates  if  necessary,  and  prepare  the  federation  for  integration  and  test. 

Step  5  -  Plan,  Integrate,  and  Test  Federation.  The  purpose  of  this  step  of  the  FEDEP  is  to  plan  the 
federation  execution,  establish  all  required  interconnectivity  between  federates,  and  test  the  federation  prior  to 
execution. 

Step  6  -  Execute  Federation  and  Prepare  Outputs.  The  purpose  of  this  step  of  the  FEDEP  is  to  execute  the 
federation  and  to  pre-process  the  output  data  from  the  federation  execution. 

Step  7  -  Analyze  Data  and  Evaluate  Results.  The  purpose  of  this  step  of  the  FEDEP  is  to  analyze  and 
evaluate  the  data  acquired  during  the  federation  execution,  and  to  support  the  results  back  to  the  user/sponsor. 
This  evaluation  is  necessary  to  assess  that  the  federation  fully  satisfies  the  requirements  of  the  user/sponsor. 
The  results  are  fed  back  to  the  user/sponsor  so  that  they  can  decide  if  the  federation  objectives  have  been  met, 
or  if  further  work  is  required. 
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4.1  TOP-LEVEL  VIEW  OF  THE  W&A  OVERLAY 

As  in  the  FEDEP,  one  of  the  design  goals  identified  during  the  development  of  this  VV&A  overlay  was  to 
define  a  highly  flexible  process.  This  facilitates  tailoring  the  VV&A  activities  defined  in  this  overlay  as  the 
user  needs  and  resource  constraints  demand.  The  actual  application  of  VV&A  techniques  to  support  the 
development  of  a  federation  could  vary  significantly  within  or  across  different  intended  uses.  However, 
the  VV&A  phases  that  correspond  to  the  FEDEP  steps  establish  a  foundation  for  applying  VV&A  within  the 
FEDEP.  Figure  4-1  illustrates  each  of  the  FEDEP  steps  together  with  the  corresponding  VV&A  overlay 
phases. 
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Figure  4-1 :  Top-Level  View  of  the  VV&A  Overlay  to  the  FEDEP. 
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The  purpose  for  each  of  the  overlay  phases  is  summarized  below: 

Phase  1  -  Verify  Federation  Objectives.  The  VV&A  Team  works  as  part  of  the  Federation  Development 
Team  to  define  a  set  of  Federation  Objectives  and  verify  the  completeness,  consistency,  and  correctness  of 
those  objectives  with  the  User/Sponsor.  The  VV&A  Team  also  assembles  the  Federation  Referent,  defines  the 
Federation  Acceptability  Criteria,  and  formulates  the  initial  Federation  Accreditation  Plan  and  V&V  Plan  at 
this  time. 

Phase  2  -  V&V  Federation  Conceptual  Model.  The  VV&A  Team  supports  the  Federation  Development 
Team  in  developing  the  Federation  Scenarios,  Federation  Conceptual  Model,  and  Federation  Requirements. 
They  also  contribute  to  verifying  these  products  and  are  responsible  for  validating  the  Federation  Conceptual 
Model. 

Phase  3  -  Verify  Federation  Design.  The  VV&A  Team  supports  selecting  the  federates  and  preparing  the 
Federation  Design.  They  also  contribute  to  verifying  the  Federation  Design  and  provide  an  updated  Federation 
Accreditation  Plan  and  V&V  Plan  for  inclusion  into  the  Federation  Development  and  Execution  Plan. 

Phase  4  -  Verify  Federation  Development  Products.  The  VV&A  Team  supports  developing  the  FOM, 
establishing  the  Federation  Agreements,  and  implementing  the  Federation  Infrastructure.  They  also  contribute 
to  verifying  the  FOM,  Federation  Agreements,  and  Federation  Infrastructure.  They  provide  any  support 
needed  to  verify  and  validate  the  federate  implementations  and  collect  the  information  from  these  V&V 
processes.  Finally,  they  verify  and  validate  the  data  sets  needed  for  federation  execution. 

Phase  5  -  Validate  and  Accept  Federation.  The  VV&A  Team  supports  the  Federation  Development  Team 
in  planning  the  federation  execution,  integrating  the  federation,  and  testing  the  federation.  They  also 
contribute  to  verifying  the  integrated  federation.  They  have  primary  responsibility  for  validating  the  results 
produced  by  the  integrated  federation  and  developing  the  federation  acceptance/accreditation 
recommendations  from  the  collected  V&V  evidence. 

Phase  6  -  V&V  Federation  Output.  The  VV&A  Team  supports  executing  the  federation  when  needed  and 
contributes  to  verifying  the  raw  execution  and  derived  output  produced  from  these  executions.  They  then 
validate  the  federation  output  when  needed. 

Phase  7  -  Consolidate  Federation  VV&A  Products.  The  VV&A  Team  supports  analyzing  the  federation 
output  when  needed  and  collects  and  assembles  the  products  from  the  preceding  VV&A  activities  into  a 
consolidated  package  to  support  future  reuse. 

The  seven  phases  in  this  overlay  parallel  the  seven  steps  in  the  FEDEP.  As  in  the  FEDEP,  this  overlay 
decomposes  each  of  the  VV&A  phases  into  a  set  of  interrelated  lower-level  activities  and  supporting  infonnation 
resources. 

4.2  OVERVIEW  OF  THE  W&A  OVERLAY  ACTIVITIES 

The  VV&A  overlay  describes  a  high  level  framework  for  the  verifying,  validating,  and  accepting  HLA 
federations.  The  intent  of  the  VV&A  overlay  is  to  specify  a  recommended  practice  that  details  the  VV&A 
phases  that  correspond  to,  support,  and  augment  the  FEDEP  steps.  Federation  stakeholders  can  leverage  this 
overlay  to  achieve  the  specific  needs  associated  with  their  intended  use. 
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Figure  4-2  illustrates  the  VV&A  overlay  and  its  interactions  with  the  FEDEP  at  equivalent  levels  of 
abstraction.  This  view  illustrates  the  flow  of  VV&A  activities,  information,  and  products  across  the  seven 
process  phases  identified  in  Figure  4-1.  Figure  4-2  depicts  the  FEDEP  steps,  the  VV&A  phases,  and  the 
primary  information  products  shared  between  these  steps  and  phases. 


Figure  4-2:  Detailed  View  of  the  W&A  Overlay  to  the  FEDEP. 


Figure  4-3  identifies  the  individual  activities  that  compose  each  of  the  VV&A  phases  that  correspond  to  the 
FEDEP’ s  seven  steps. 
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Figure  4-3:  Activities  Associated  with  Each  W&A  Overlay  Phase. 


Although  many  of  the  activities  represented  in  this  overlay  diagram  appear  highly  sequential,  the  intention  is  not 
to  suggest  a  strict  waterfall  approach  to  W&A.  Rather,  this  process  illustration  is  simply  intended  to  highlight 
the  major  W&A  activities  that  occur  during  federation  development  and  execution  and  approximately  when 
such  activities  are  first  initiated  relative  to  other  VV&A  or  FEDEP  activities. 
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Chapter  5  -  CONCLUSIONS  AND  RECOMMENDATIONS 

When  MSG-0 19/TG-0 16  was  established,  both  the  NATO  and  the  distributed  simulation  communities  had 
realized  a  need  for  guidance  about  the  VV&A  of  federations.  The  efforts  of  this  Task  Group  addressed  this 
need  for  both  communities.  In  the  course  of  addressing  the  common  need  for  federation  VV&A,  the  Task 
Group  began  to  develop  a  baseline  view  of  VV&A.  This  view  emphasized  the  lack  of  a  universal  model  for 
federation  VV&A  processes  due  to  divergent  national  perspectives  and  needs.  The  Task  Group  determined 
that  the  lack  of  this  consistent  methodology  was  hampering  NATO’s  ability  to  develop  shared  simulation 
federations.  Resolving  these  disparities  required  developing  a  consistent  understanding  of  what  VV&A  is  and 
how  to  apply  it  to  achieve  the  best  results.  Even  though  the  member  nations  came  from  different  perspectives, 
they  were  able  to  converge  onto  a  single  unified  model  of  federation  VV&A  processes.  Thus,  the  baseline  and 
the  results  from  the  CConVV&A  [21]  resulted  in  a  decision  to  merge  the  efforts  of  this  Task  Group  and  the 
SISO  VV&A  Overlay  drafting  group.  This  merge  expanded  the  focus  of  this  Task  Group  to  go  beyond 
preparing  a  technical  report  to  actually  producing  a  VV&A  Overlay  standard  product.  The  Task  Group  also 
identified  several  terms  needed  to  describe  VV&A  processes  for  federations  and  developed  definitions  for 
these  terms  using  existing  standard  definitions. 

Formalizing  the  Overlay  as  an  IEEE  standard  was  determined  to  be  the  most  appropriate  standardization 
approach  as  it  provides  international  recognition  and  access;  co-locates  the  Overlay  with  its  companion 
document  IEEE  standard  1516.3  for  the  HLA  FEDEP  [20];  and  provides  maintenance  and  management  of  the 
Overlay  by  IEEE.  The  proposed  IEEE  standard  (1516.4)  addresses  the  needs  of  both  the  NATO  and  SISO 
communities  for  a  VV&A  Overlay  to  the  FEDEP. 

This  Task  Group  succeeded  because  of  extensive  collaboration  with  other  international  groups  including 
SISO,  REVVA  and  ITOP.  The  tight  coupling  between  the  NATO  and  SISO  groups  resulted  in  a  better,  more 
widely  applicable  Overlay  (e.g.,  government,  industry,  and  academia).  The  collaboration  benefited  NATO  by 
directly  coupling  the  MSG  product  to  an  internationally  recognized  standards  body  and  benefited  SISO  by 
ensuring  early  and  extensive  international  input  to  the  final  product.  This  collaborative  effort  could  serve  as  a 
model  for  future  joint  M&S  standards  initiatives. 

While  the  Task  Group  feels  that  enormous  strides  have  been  made  with  the  development  of  the  VV&A 
Overlay,  still  more  work  is  needed  to  increase  the  effectiveness  and  efficiency  of  the  VV&A  processes. 
Particular  areas  to  be  addressed  include  case  studies  to  mature  the  Overlay,  tailoring  the  VV&A  processes 
based  on  user  risk  management,  and  effective  communication  of  risk  to  program  management. 

Based  on  the  above  conclusions,  the  Task  Group  makes  the  following  recommendations: 

•  NATO  should  adopt  Appendix  1  as  an  interim  Standardization  Agreement  (STANAG)  until  the  IEEE 
15 16.4  standard  is  issued  and  then  revise  the  STANAG  to  adopt  the  IEEE  standard. 

•  MSG-054,  the  follow-on  study  group,  should: 

•  Participate  in  the  IEEE  1516.4  balloting  process; 

•  Monitor  on-going  VV&A  case  study  initiative  and  incorporate  suggested  improvements  to  the 
Overlay  during  the  balloting  process;  and 

•  Study  the  relationship  between  user  risk,  acceptance  criteria,  and  tailoring  of  the  VV&A  process. 

•  NATO  should  consider  adopting  similar  collaborative  relationships  with  international  standards 
bodies  as  appropriate. 
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Acceptability  Criteria:  The  criteria  that  the  federation  needs  to  meet  to  be  acceptable  for  its  intended  use. 

Acceptance:  The  decision  to  use  the  results  produced  by  a  federation  for  an  intended  use. 

Accreditation:  The  official  certification  that  a  model  or  simulation  is  acceptable  for  use  for  a  specific 
purpose. 

Activity:  A  set  of  tasks  that  consumes  time  and  resources  and  whose  performance  is  necessary  for  the 
execution  of  the  federation  development  and  execution  process. 

Federate:  An  application  that  may  be  or  is  currently  coupled  with  other  software  applications  under  a  Federation 
Object  Model  Document  Data/Federation  Execution  Data  and  a  runtime  infrastructure  (e.g.,  federation 
managers,  data  collectors,  real  world  (live)  systems  (e.g.,  Command,  Control,  Communications,  Computers,  and 
Intelligence  (C4I)  systems,  instrumented  ranges,  sensors),  simulations,  passive  viewers,  and  other  utilities). 

Federation:  A  named  set  of  federate  applications  and  a  common  Federation  Object  Model  (FOM)  that  are 
used  as  a  whole  to  achieve  some  specific  objective. 

Federation  Conceptual  Model:  An  abstraction  of  the  real  world  that  serves  as  a  frame  of  reference  for 
federation  development  by  documenting  simulation-neutral  views  of  important  entities  and  their  key  actions 
and  interactions. 

Federation  Object  Model:  A  specification  defining  the  information  exchanged  at  runtime  to  achieve  a  given 
set  of  federation  objectives. 

Federation  Objectives:  The  statement  of  the  problem  that  is  to  be  addressed  by  the  establishment  and 
execution  of  a  federation. 

Federation  Requirements:  A  statement  that  identifies  a  federation  characteristic,  constraint,  process, 
or  product  that  is  unambiguous  and  testable  and  that  is  necessary  for  a  federation  to  be  acceptable  for  its 
intended  use. 

Federation  Scenario:  a  set  of  initial  conditions  and  time  line  of  significant  events  used  within  a  federation 
execution  to  achieve  federation  objectives. 

Referent:  A  codified  body  of  knowledge  about  a  thing  being  simulated. 

Validation:  The  process  of  evaluating  a  federation  throughout  the  development  and  execution  process  to 
determine  how  well  it  satisfies  the  acceptability  criteria  within  the  context  of  the  referent. 

Verification:  The  process  of  evaluating  a  federation  and  its  intermediate  products  to  determine  whether  the 
products  from  a  given  development  phase  satisfy  the  conditions  imposed  at  the  start  of  that  phase  and, 
ultimately,  determining  that  an  implementation  of  a  federation  correctly  and  completely  represents  the 
developer’s  conceptual  description  and  specifications. 
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C4I 

Command,  Control,  Communications,  Computers,  and  Intelligence 

CConVV&A 

Combined  Convention  on  International  VV&A  Standardization  Endeavours 

DG(SRA) 

Director  General  (Scrutiny  &  Analysis)  (UK) 

DMSO 

Defense  Modeling  &  Simulation  Office  (US) 

DoD 

Department  of  Defense  (US) 

Dstl 

Ministry  of  Defence,  Defence  Science  and  Technology  Laboratory  (UK) 

Euro- SI  W 

European  Simulation  Interoperability  Workshop 

FEDEP 

Federation  Development  and  Execution  Process 

FMV 

Forsvarets  materielverk  (Sweden) 

FOM 

Federation  Object  Model 

HLA 

High  Level  Architecture 

IEEE 

Institute  of  Electrical  and  Electronics  Engineers 

ITEC 

International  Training  and  Education  Conference 

ITOP 

International  Test  Operations  Procedures 

M&S 

Modelling  &  Simulation 

MSG-019 

Modelling  &  Simulation  Group  019 

MSG-054 

Modelling  &  Simulation  Group  054 

NATO 

North  Atlantic  Treaty  Organisation 

NAVSEA 

Naval  Sea  Systems  Command  (US) 

NDAT 

NAVSEA  Dahlgren  Accreditation  Team  (US) 

NMSG 

NATO  Modelling  and  Simulation  Group 

PDG 

Product  Development  Group 

REVVA 

WEAG  Research  and  Technology  program  on  VV&A  (JP1 1.20) 

RPG 

Recommended  Practices  Guide  (US) 

SISO 

Simulation  Interoperability  Standards  Organization 

SIW 

Simulation  Interoperability  Workshop 

STANAG 

Standardization  Agreement 
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TG 

Task  Group 

UBM 

Universitat  der  Bundeswehr  Miinchen  (Germany) 

V&V 

Verification  and  Validation 

VV&A 

Verification,  Validation,  &  Accreditation 

WEAG 

Western  European  Armament  Group 
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Draft  Recommended  Practice  for  Verification,  Validation,  and  Accreditation 
of  a  Federation,  an  Overlay  to  the  High  Level  Architecture  Federation 
Development  and  Execution  Process 


1.0  OVERVIEW 

1.1  Scope 

This  recommended  practice  defines  the  processes  and  procedures  that  should  be  followed  to  implement 
Verification,  Validation,  and  Accreditation1  (VV&A)  for  federations  being  developed  using  the  High  Level 
Architecture  (HLA)  Federation  Development  and  Execution  Process  (FEDEP).  This  recommended  practice  is 
not  intended  to  replace  existing  VV&A  policies,  procedures,  and  guidance,  but  rather  is  intended  to  focus  on 
the  unique  aspects  of  the  VV&A  of  federations.  It  provides  a  higher-level  framework  into  which  such 
practices  can  be  integrated  and  tailored  for  specific  uses.  The  VV&A  overlay  provides  implementation-level 
guidance  to  VV&A  practitioners;  however,  it  does  not  describe  the  individual  techniques  that  might  be 
employed  to  execute  the  VV&A  processes  for  federations. 

This  VV&A  overlay  focuses  upon  the  VV&A  processes  that  apply  to  federations  and  not  the  VV&A 
processes  associated  with  individual  simulations  (federates),  but  does  consider  using  the  information  produced 
by  those  processes. 

Users,  developers,  and  VV&A  personnel  working  with  simulations  and  simulation  compositions  not  based 
upon  the  HLA  and  the  FEDEP  can  also  benefit  from  the  guidance  in  this  document  since  the  activities  that  this 
overlay  describes  can  be  tailored  to  support  any  type  of  distributed  simulation  application. 

1.2  Purpose 

The  VV&A  overlay  has  been  designed  to  apply  across  a  wide  range  of  functional  applications.  The  purpose  of 
this  overlay  is  to  provide  a  more  detailed  view  of  the  VV&A  processes  implied  by  the  FEDEP.  Currently, 
these  processes  represent  the  best  practices  available  to  the  VV&A  community.  The  VV&A  overlay  is  a 
tailorable  process  and  is  offered  as  guidance  to  all  participants  in  FEDEP  activities. 

This  overlay  identifies  and  describes  the  recommended  VV&A  processes  that  should  be  followed  to  assure  the 
acceptability  and  utility  of  federations  for  particular  intended  uses.  The  overlay  also  identifies  and  describes 
the  information  feeding  and  resulting  from  those  processes  as  well  as  the  relationships  between  the  FEDEP 
and  the  VV&A  processes  and  their  respective  information  products.  In  addition,  this  overlay  defines  those 
terms  uniquely  needed  to  characterize  the  FEDEP  VV&A  overlay.  This  overlay  takes  special  care  to  use  and 
build  upon  existing  standards,  standard  terms  and  their  definitions  whenever  possible. 


1  Note  that  outside  of  the  United  States  there  may  not  be  a  formal  accreditation  process  and  the  terns  “acceptance”  or  “accepted  for 
use”  may  be  used;  in  this  document  the  term  acceptance  is  the  decision  to  use  a  model,  simulation,  or  federation  of  models  and 
simulations  for  a  specific  purpose  and  the  term  accreditation  is  the  official  certification  that  a  model,  simulation,  or  federation  of 
models  and  simulations  is  acceptable  for  use  for  a  specific  purpose.  For  the  purposes  of  this  document  the  terms  are  equivalent. 
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1.3  Conventions 

This  section  describes  the  conventions  adopted  for  presenting  the  overlay  material.  The  overlay  employs  three 
types  of  conventions:  capitalization,  special  words,  and  graphic  symbols.  These  conventions  are  used  throughout 
the  overlay  as  much  as  possible. 

The  overlay  uses  capitalization  to  identify  the  key  participants  in  the  VV&A  processes.  The  overlay 
distinguishes  between  the  Federation  Development  Team  and  the  VV&A  Team,  and  capitalizes  the  references 
to  these  teams.  The  VV&A  Team  may  participate  as  part  of  the  Federation  Development  Team.  The  VV&A 
Team  may  participate  as  part  of  the  Federation  Development  Team;  however,  the  VV&A  Team  may  or  may 
not  be  organizationally  independent  of  it. 

The  titles  of  the  primary  information  produced  or  needed  by  these  teams  are  also  capitalized  (e.g.,  Federation 
Objectives,  Federation  Acceptability  Criteria,  and  Federation  Referent).  This  information  includes  that 
produced  by  both  the  Federation  Development  Team  (e.g.,  Federation  Objectives  and  Federation  Conceptual 
Model)  and  the  VV&A  Team  (e.g.,  Federation  Acceptability  Criteria  and  Federation  Referent).  Any  modifiers 
describing  those  information  items  are  not  capitalized  (e.g.,  initial  Federation  V&V  Plan,  revised  Federation 
V&V  Plan,  validated  Federation  Conceptual  Model)  as  part  of  the  title.  These  capitalization  conventions 
emphasize  the  content  of  the  information  produced. 

The  overlay  uses  a  few  words  in  very  specific  ways.  It  divides  the  VV&A  processes  applied  to  federations 
into  phases,  activities,  and  tasks.  Phases  correspond  to  the  seven  top-level  FEDEP  steps.  Each  phase  contains 
several  activities  and  each  activity  includes  a  set  of  tasks  that  the  VV&A  Team  executes  to  accomplish  the 
activity. 

The  overlay  also  distinguishes  between  the  three  types  of  VV&A  Team  participation  in  the  FEDEP: 

•  Supporting; 

•  Contributing  to;  and 

•  Responsible  for  performing. 

The  overlay  consistently  begins  both  activity  titles  and  task  descriptions  with  the  verbs  corresponding  to  these 
types  and  uses  this  convention  throughout  the  document.  This  distinction  occurs  because  of  the  many  different 
players  in  the  FEDEP  and  the  variety  of  ways  in  which  various  responsibilities  may  be  assigned  to  these 
players. 

All  activity  titles  and  task  descriptions  that  begin  with  the  verb  “support”  represent  those  activities  or  tasks 
that  the  FEDEP  explicitly  identifies  and  that  the  W&A  Team  can  support  in  some  way.  Activities  with 
“support”  titles  are  generally  led  or  performed  by  a  member  of  the  Federation  Development  Team,  but  require 
the  involvement  of  the  W&A  Team.  A  “support”  activity  may  contain  tasks  that  the  FEDEP  does  not 
explicitly  identify.  The  VV&A  Team  performs  these  tasks  and  assumes  the  responsibility  for  satisfactorily 
completing  them. 

The  activity  titles  and  task  descriptions  that  begin  “contribute  to”  represent  verification  activities  and  tasks  and 
need  to  be  accomplished  to  ensure  the  success  of  the  VV&A  effort.  As  a  result,  the  VV&A  Team  may  need  to  be 
involved  in  performing  these  activities  and  tasks,  but  will  generally  not  take  primary  responsibility  for  their 
completion.  However,  this  does  not  imply  that  the  W&A  Team  does  not  perfonn  verification  activities.  On  the 
contrary,  the  W&A  Team  may  need  to  perform  verification  where  they  need  the  information  to  supplement  that 


A1  -2 


RTO-TR-MSG-019 


APPENDIX  1  -  VV&A  OVERLAY  TO  THE  FEDEP 


received  from  the  verification  that  other  members  of  the  Federation  Development  Team  have  performed.  In 
some  cases,  the  W&A  Team  may  need  to  assume  responsibility  for  performing  the  “contribute  to”  activities 
and  tasks  when  that  responsibility  has  been  assigned  to  no  other  member  of  the  Federation  Development  Team. 
This  may  be  needed  so  that  the  VV&A  Team  can  perform  the  duties  for  which  they  do  have  explicit 
responsibility. 

Those  activity  titles  and  task  descriptions  that  begin  with  some  other  verb  (e.g.,  plan,  validate,  perform, 
prepare)  represent  the  activities  and  tasks  for  which  the  VV&A  Team  has  primary  responsibility  and  will 
perform.  These  activities  and  tasks  are  essential  to  creating  a  valid  federation.  Consistent  with  these 
conventions,  these  VV&A-specific  tasks  do  not  begin  with  either  the  “support”  or  “contribute  to”  verbs. 

Each  phase  described  in  this  overlay  has  an  activity  diagram  associated  with  it  that  shows  the  primary 
relationships  between  the  VV&A  activities  in  the  phase  and  the  activities  in  the  corresponding  FEDEP  step. 
These  diagrams  use  particular  symbols  to  identify  the  types  of  VV&A  activities  and  their  interactions. 
The  Table  Al-1  below  describes  the  meanings  of  these  symbols. 

Table  Al-1:  Symbol  Conventions  Used  in  the  Activity  Diagrams 


Symbol 

Meaning 

C 

A  FEDEP  activity 

- - - 

An  activity  that  directly  supports  a  corresponding  FEDEP 
activity 

\  / 

An  activity  to  which  the  VV&A  Team  contributes  and  may 
perform  if  needed 

An  activity  that  the  VV &ATeam  uniquely  performs,  leads, 
and  takes  responsibility  for  its  satisfactory  completion 

Data  sets  needed  to  support  a  FEDEP  execution 

- ► 

A  prim  ary  information  flow  from  one  activity  to  another  or 
from  one  VV  &A  phase  to  another 

1 - * 

A  connection  between  two  information  flows 

These  activity  diagrams  label  each  VV&A  activity  with  a  number  designation  (X,Y)  to  show  the  traceability 
between  the  phase  (X)  with  which  the  activity  is  associated  and  the  activity  (Y).  The  activity  numbers  in  these 
diagrams  are  only  identifiers  and  do  not  imply  any  ordering  of  activity  execution. 

These  activity  diagrams  present  only  abstractions  of  the  details  provided  in  the  activity  discussions. 
These  diagrams  are  intended  only  as  illustrations  to  clarify  the  key  elements  of  the  phases  they  described. 
The  activity  diagrams  show  only  the  primary  information  exchanged  and  do  not  include  all  of  the  information 
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involved  in  the  activity  interactions.  The  information  illustrated  in  these  diagrams  either  identifies  what  is 
important  to  the  VV&A  effort  or  emphasizes  the  value  added  by  the  VV&A  activities.  The  activity 
discussions  provide  more  extensive  lists  of  the  information  inputs  and  outputs.  The  abstraction  inherent  to 
these  diagrams  is  not  intended  to  imply  that  these  activities  do  not  draw  upon  information  prepared  in  the 
preceding  phases.  The  conventions  for  these  diagrams  are  informal,  do  not  adhere  to  the  specifications  of 
formal  diagramming  conventions  and  are  provided  only  for  illustrative  purposes. 


2.0  REFERENCES 

The  three  specifications  that  compose  the  HLA,  together  with  the  FEDEP,  provide  the  technical  foundation 
for  designing  and  developing  all  HLA  federations.  These  specifications  are  described  in  the  following 
documents: 

IEEE  Std  151 6™-2000,  IEEE  Standard  for  Modeling  and  Simulation  (M&S)  High  Level  Architecture 
(HLA)  -  Framework  and  Rules. 

IEEE  Std  1516.1™-2000,  IEEE  Standard  for  Modeling  and  Simulation  (M&S)  High  Level  Architecture 
(HLA)  -  Federate  Interface  Specification. 

IEEE  Std  1516.2™-2000,  IEEE  Standard  for  Modeling  and  Simulation  (M&S)  High  Level  Architecture 
(HLA)  -  Object  Model  Template  (OMT)  Specification. 

IEEE  Std  1516.3™-2003  IEEE  Recommended  Practice  For  High  Level  Architecture  (HLA)  Federation 
Development  and  Execution  Process  (FEDEP). 

(The  IEEE  products  referred  to  in  this  clause  are  trademarks  belonging  to  the  Institute  of  Electrical  and 
Electronics  Engineers,  Inc.). 


3.0  DEFINITIONS,  ACRONYMS,  AND  ABBREVIATIONS 

The  following  terms  and  definitions  apply  to  this  recommended  practice.  The  Authoritative  Dictionary  of 
IEEE  Standards,  Seventh  Edition,  should  be  referenced  for  terms  not  defined  in  this  clause. 

3.1  Definitions 

Acceptability  Criteria:  The  criteria  that  the  model,  simulation,  or  federation  of  models  and  simulations  needs 
to  meet  to  be  acceptable  for  its  intended  use. 

Acceptance:  The  decision  to  use  the  results  produced  by  a  model,  simulation,  or  federation  of  models  and 
simulations  for  an  intended  use. 

Accreditation:  The  official  certification  that  a  model,  simulation,  or  federation  of  models  and  simulations  is 
acceptable  for  use  for  a  specific  purpose  (see  DoDI  5000.61). 

Activity:  A  set  of  tasks  that  consumes  time  and  resources  and  whose  performance  is  necessary  for  the 
execution  of  the  federation  development  and  execution  process. 

Credibility:  The  belief  that  a  model,  simulation,  or  federation  of  models  and  simulations  can  serve  an 
intended  use. 
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Error  Characteristics:  Those  characteristics  that  describe  the  nature  of  the  errors  in  the  output  from  a  model, 
simulation,  or  a  federation  of  models  and  simulations.  For  each  output  variable  that  can  assume  metric  values, 
the  minimum  and  maximum  error  bounds  are  measured  from  the  values  established  by  the  referent  under  the 
same  conditions  together  with  the  probability  that  all  errors  of  the  output  values  for  that  variable  will  fall 
within  those  error  bounds. 

Federate:  An  application  that  may  be  or  is  currently  coupled  with  other  software  applications  under  a 
Federation  Object  Model  Document  Data/Federation  Execution  Data  and  a  runtime  infrastructure 
(e.g.,  federation  managers,  data  collectors,  real  world  (live)  systems  (e.g.,  C4I  systems,  instrumented  ranges, 
sensors),  simulations,  passive  viewers,  and  other  utilities)  (see  IEEE  1516.3™-2003). 

Federation:  A  named  set  of  federate  applications  and  a  common  Federation  Object  Model  (FOM)  that  are 
used  as  a  whole  to  achieve  some  specific  objective  (see  IEEE  1516.3™-2003). 

Federation  Conceptual  Model:  An  abstraction  of  the  real  world  that  serves  as  a  frame  of  reference  for 
federation  development  by  documenting  simulation-neutral  views  of  important  entities  and  their  key  actions 
and  interactions  (see  IEEE  1516.3™-2003). 

Federation  Object  Model:  A  specification  defining  the  information  exchanged  at  runtime  to  achieve  a  given 
set  of  federation  objectives  (see  IEEE  1516.3™-2003). 

Federation  Objectives:  The  statement  of  the  problem  that  is  to  be  addressed  by  the  establishment  and 
execution  of  a  federation  (see  DoD  5000. 59-M). 

Federation  Requirements:  A  statement  that  identifies  a  federation  characteristic,  constraint,  process,  or  product 
that  is  unambiguous  and  testable  and  that  is  necessary  for  a  federation  to  be  acceptable  for  its  intended  use. 

Federation  Scenario:  a  set  of  initial  conditions  and  time  line  of  significant  events  used  within  a  federation 
execution  to  achieve  federation  objectives. 

Fidelity:  The  description  of  a  model,  simulation,  or  federation  of  models  and  simulations  and  its  associated 
data  representational  capabilities  (e.g.,  resolution,  error,  precision,  sensitivity). 

Interoperability:  The  ability  of  a  federate  to  provide  services  to  and/or  accept  services  from  other  federates 
and  to  use  the  services  so  exchanged  to  enable  the  federates  to  operate  effectively  together. 

Referent:  A  codified  body  of  knowledge  about  a  thing  being  simulated  (see  Verification,  Validation, 
and  Accreditation  (VV&A)  Recommended  Practices  Guide,  Build  2.5). 

Representational  Requirements:  That  subset  of  the  modeling  and  simulation  requirements  that  specifically 
describes  the  required  states  or  behavior  of  the  things  that  the  simulation  represents,  including  the  modeled 
entities,  their  properties,  and  their  dependencies  (see  Verification,  Validation,  and  Accreditation  (VV&A) 
Recommended  Practices  Guide,  Build  2.5). 

Results  Sampling  Strategy:  The  approach  for  collecting  the  output  from  a  model,  simulation,  or  federation  of 
models  and  simulations. 

Risk:  A  measure  of  the  probability  and  severity  of  undesired  effects  often  taken  as  the  simple  product  of 
probability  and  consequence  (see  The  Authoritative  Dictionary  of  IEEE  Standards  Terms,  Seventh  Edition). 
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Traceability:  The  degree  to  which  a  relationship  can  be  established  between  two  or  more  products  of  the 
development  process,  especially  products  having  a  predecessor-successor  or  master-subordinate  relationship 
to  one  another;  for  example,  the  degree  to  which  the  requirements  and  design  of  a  given  system  element  match 
(see  The  Authoritative  Dictionary  of  IEEE  Standards  Terms,  Seventh  Edition). 

Uncertainty:  A  quantity  or  quantities  that  indicate  the  degree  of  doubt  associated  with  the  value  of  a 
simulated  or  referent  property;  for  enumerated  values,  the  probability  that  the  actual  enumerated  value 
corresponds  with  the  simulated  or  referent  value  under  the  same  conditions;  for  metric  values,  the  error 
characteristics  associated  with  the  simulated  or  referent  value. 

Validation:  The  process  of  evaluating  a  model,  simulation,  or  federation  of  models  and  simulations 
throughout  the  development  and  execution  process  to  determine  how  well  it  satisfies  the  acceptability  criteria 
within  the  context  of  the  referent. 

Validity:  The  property  of  a  model,  simulation,  or  federation  of  models  and  simulations  representations  being 
complete  and  correct  enough  for  the  intended  use. 

Verification:  The  process  of  evaluating  a  model,  simulation,  or  federation  of  models  and  simulations  and  its 
intermediate  products  to  determine  whether  the  products  from  a  given  development  phase  satisfy  the 
conditions  imposed  at  the  start  of  that  phase  and,  ultimately,  determining  that  an  implementation  of  a  model, 
simulation,  or  federation  of  models  and  simulations  correctly  and  completely  represents  the  developer’s 
conceptual  description  and  specifications. 

3.2  Acronyms  and  Abbreviations 

FEDEP  Federation  Development  and  Execution  Process 

FOM  Federation  Object  Model 

HLA  High  Level  Architecture 

IEEE  Institute  of  Electrical  and  Electronics  Engineers 

M&S  Modeling  and  Simulation 

SOM  Simulation  Object  Model  (of  a  federate) 

V&V  Verification  and  Validation;  Verify  and  Validate 

VV&A  Verification,  Validation,  and  Accreditation 

4.0  FEDERATION  W&A  ROLES  AND  RESPONSIBILITIES 

This  section  introduces  the  various  roles  and  responsibilities  that  are  required  to  perform  each  activity 
identified  in  the  VV&A  Overlay.  The  person  or  persons  who  perform  in  the  roles  needed  to  accomplish  the 
activities  in  this  overlay  comprise  the  VV&A  Team.  The  section  also  describes  the  rationale  used  to  identify 
and  classify  these  roles  and  responsibilities. 

The  roles  and  responsibilities  defined  are  not  intended  to  map  directly  to  specific  people  or  organizations. 
There  are  cases  where  one  individual  or  organization  may  have  several  of  the  roles.  The  delegation  of  these 
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responsibilities  depends  upon  the  available  resources.  The  resources  required  to  support  each  of  these  roles 
depends  on  the  User/Sponsor  requirements  and  the  complexity  of  the  federates  and  federations.  If  these 
resources  are  not  available,  then  some  prioritization  and  tailoring  of  the  tasks  and  their  assignment  to  the  roles 
will  be  necessary. 

Table  A 1-2  identifies  the  roles  and  responsibilities  relevant  to  VV&A  for  each  activity  in  this  overlay. 
They  are  intended  only  as  general  guidelines.  Interpreting  Table  Al-2  requires  understanding  the  actions 
attributed  to  the  roles  and  the  heuristics  used  to  assign  the  actions  to  the  roles. 


The  following  actions  are  used  in  Table  Al-2  and  in  the  heuristics  used  to  construct  that  table.  They  can  apply 
to  either  an  organization  or  an  individual: 


•  Lead  (L) 

•  Perform  (P) 

•  Assist  (AST) 

•  Review  (R) 

•  Monitor  (M) 

•  Approve  (AP) 


Responsible  for  the  activity  including  assignment  of  actions  to  others  and  scheduling. 

Responsible  for  performing  an  activity  or  task  under  the  direction  of  the  leader  of  the 
activity. 

Help  a  performer  or  a  leader  complete  an  activity  or  task. 

Called  upon  to  review  the  work  of  others  and  to  make  suggestions  for  improvement 
where  appropriate. 

Will  watch  the  activities  being  performed  or  receive  the  products  so  that  they  can  be 
aware  of  the  VV&A  status. 

Responsible  for  approving  the  VV&A-related  products. 


The  following  heuristics  were  used  to  develop  Table  Al-2: 

•  Federation  related  (non  VV&A)  activities  are  led  or  performed  by  the  Federation  Manager  or  Federation 
Developer; 

•  Verification  activities  may  be  led  or  performed  by  the  Federation  Developer  and  assisted  by  the  V&V 
Agent;  or  led  and  performed  by  the  V&V  Agent,  if  needed; 

•  Validation  activities  are  led  or  performed  by  the  V&V  Agent; 

•  Accreditation  activities  are  led  or  performed  by  the  Accreditation  Agent; 

•  For  “support”  activities,  the  V&V  Agent  or  the  Accreditation  Agent  generally  monitor  or  assist;  and 

•  For  “contribute  to”  activities,  the  V&V  Agent  or  the  Accreditation  Agent  may  either  assist  or  perform. 
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Table  A1-2:  Roles  and  Responsibilities 


W&A  Overlay  Phases  and  Activities 

L  =  Lead.  P  =  Perform.  AST  =  Assist. 

R  =  Review,  M  =  Monitor.  AP  =  Approve 

User  / 
Sponsor 

Federation 

Manager 

Federation 

Developer 

V&V 

Agent 

Accreditation 

Agent 

Subject 

Matter 

Expert 

Phase  1  -  Verify  federation  objectives 

Activity  1.1 

Support  identifying  user/sponsor  needs 

AST,  AP 

L,  P 

AST 

M 

AST 

AST 

Activity  1.2 

Plan  accreditation  activities 

R,  AP 

AST 

AST 

L,  P 

AST 

Activity  1.3 

Support  developing  federation  objectives 

AST,  AP 

L,  P 

AST 

M 

AST 

AST 

Activity  1.4 

Contribute  to  verifying  federation  objectives 

AP 

L 

P 

AST 

AST 

AST 

Activity  1.5 

Assemble  federation  referent 

AST,  AP 

M 

AST,  P 

AST 

L,  P 

AST 

Activity  1.6 

Define  federation  acceptability  criteria 

AST,  AP 

M 

AST 

AST 

L,  P 

AST 

Activity  1.7 

Plan  V&V  activities 

R 

AST,  AP 

R 

L,  P 

AST 

AST 

Phase  2  -  V&V  federation  conceptual  model 

Activity  2.1 

Support  developing  federation  scenarios 

AP 

L 

P 

AST 

M 

P,  AST 

Activity  2.2 

Contribute  to  verifying  federation  scenarios 

M 

L,  P 

AST 

M 

P,  AST 

Activity  2.3 

Support  developing  federation  conceptual  model 

AP 

L 

P 

AST 

M 

P,  AST 

Activity  2.4 

Contribute  to  verifying  federation  conceptual  model 

M 

L,  P 

AST 

M 

P,  AST 

Activity  2.5 

Validate  federation  conceptual  model 

AP 

M 

AST 

L,  P 

M 

P,  AST 

Activity  2.6 

Support  developing  federation  requirements 

AST,  AP 

L,  P 

AST 

AST 

M 

AST 

Activity  2.7 

Contribute  to  verifying  federation  requirements 

AP 

L 

P 

AST 

M 

AST 

Phase  3  -  Verify  federation  design 

Activity  3.1 

Support  selecting  federates 

AP 

L 

P 

AST 

M 

AST 

Activity  3.2 

Support  preparing  federation  design 

AP 

L 

P 

AST 

M 

AST 

Activity  3.3 

Contribute  to  verifying  federation  design 

M 

L.  P 

AST 

M 

AST 

Activity  3.4 

Support  preparing  federation  development  and  execution 
plan 

AP 

L 

P 

AST 

AST 

AST 
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5.0  OVERLAY  ASSUMPTIONS  AND  TAILORING 

5.1  Assumptions 

The  following  assumptions  were  made  during  development  of  the  VV&A  overlay  to  scope  the  responsibilities 
of  the  VV&A  Team.  They  have  been  arranged  loosely  in  logical,  not  priority,  order: 

•  The  Federation  Development  Team  will  follow  the  FEDEP  in  their  development  and  execution  of 
their  federation. 

•  Coordination  and  cooperation  between  the  VV&A  Team  and  the  Federation  Development  Team  is 
essential. 

•  The  Federation  Development  Team  leads  and  performs  federation  verification  activities. 

•  The  VV&A  Team  leads  and  performs  federation  validation  activities. 

•  Federate  V&V  information  and  use  histories  can  provide  valuable  information  for  federation  VV&A. 

•  The  VV&A  Team  is  not  responsible  for  validating  or  verifying  the  individual  federates  although  they 
do  rely  upon  the  products  of  these  activities  and  may  request  or  perform  supplemental  verification 
and  validation  to  gain  the  information  they  need  for  federation  VV&A. 

•  The  VV&A  Team  will  have  timely  access  to  the  information  they  require. 

•  The  User/Sponsor  will  agree  that  a  finite  set  of  observable  or  measurable  acceptability  criteria  will  be 
used  to  judge  federation  acceptability  for  an  intended  use. 

•  The  VV&A  processes  may  be  executed  iteratively  throughout  the  federation  development  and 
execution  process. 

•  All  federations  will  be  subject  to  an  acceptance  process  that  may  or  may  not  include  accreditation. 

•  If  needed,  accreditation  of  the  federation  can  occur  anywhere  from  Activity  5.5  through  Activity  7.1. 

•  The  federation  VV&A  processes  described  in  this  overlay  can  and  should  be  tailored  to  meet  the 
needs  of  the  individual  applications. 

5.2  Tailoring 

The  VV&A  activities  described  in  this  document,  while  being  generally  applicable  to  most  HLA  federations, 
are  intended  to  be  tailored  to  meet  the  needs  of  each  individual  application.  Every  federation  application  is 
unique.  The  extent  to  which  these  VV&A  processes  can  be  performed  for  a  given  federation  application  will 
depend  on  a  number  of  factors,  including  the  quality  of  the  requirements  information  and  the  resources 
allocated  to  the  VV&A  Team.  The  recommended  practices  provided  in  this  document  should  be  used  as  a 
starting  point  for  developing  the  specific  approach  to  federation  VV&A  needed  to  support  the  intended  use. 

Below  are  some  perspectives  on  tailoring  the  VV&A  processes  to  best  fit  the  circumstances  of  each  individual 
application. 

•  Tailoring  is  driven  by  risk  and  resource  constraints  (e.g.,  available  information,  budget,  skills,  time). 

•  Quantifying  and  minimizing  risk  are  factors  that  drive  tailoring  decisions. 

•  The  correctness  and  completeness  of  the  federation  VV&A  products  can  reduce  the  risk  incurred  by 
using  the  federation  results  (i.e.,  use  risk). 
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*  The  absence  of  certain  information  input  to  the  VV&A  processes  will  limit  the  quality  of  the  content 
of  the  VV&A  products. 

*  Tailoring  decisions  have  an  impact  upon  use  risk  and  the  resources  required,  and  this  impact  should 
be  understood  and  documented. 

*  The  VV&A  processes  should  leverage  testing  and  data  collection  tasks  and  describe  this  in  the 
Federation  Accreditation  Plan  and  V&V  Plan. 

*  Any  tailoring  should  adhere  to  any  existing  policies,  standards  or  guidelines  that  may  be  relevant  to 
the  intended  use. 

*  Tailoring  may  be  done  at  any  time  in  the  federation  VV&A  processes. 


6.0  VV&A  OVERLAY  MODEL:  TOP-LEVEL  VIEW 

As  in  the  FEDEP,  one  of  the  design  goals  identified  during  the  development  of  this  VV&A  overlay  was  to 
define  a  highly  flexible  process.  This  facilitates  tailoring  the  VV&A  activities  defined  in  this  overlay  as  the 
user  needs  and  resource  constraints  demand.  The  actual  application  of  VV&A  techniques  to  support  the 
development  of  a  federation  could  vary  significantly  within  or  across  different  intended  uses.  Flowever, 
the  VV&A  phases  that  correspond  to  the  FEDEP  steps  establish  a  foundation  for  applying  VV&A  within  the 
FEDEP.  Figure  Al-1  illustrates  each  of  the  FEDEP  steps  together  with  the  corresponding  VV&A  overlay 
phases. 
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Figure  Al-1:  Top-Level  View  of  the  VV&A  Overlay  to  the  FEDEP. 


The  purpose  for  each  of  the  overlay  phases  is  summarized  below: 

•  Phase  1:  Verify  Federation  Objectives  -  The  W&A  Team  works  as  part  of  the  Federation 
Development  Team  to  define  a  set  of  Federation  Objectives  and  verify  the  completeness,  consistency, 
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and  correctness  of  those  objectives  with  the  User/Sponsor.  The  VV&A  Team  also  assembles  the 
Federation  Referent,  defines  the  Federation  Acceptability  Criteria,  and  formulates  the  initial 
Federation  Accreditation  Plan  and  V&V  Plan  at  this  time. 

•  Phase  2:  V&V  Federation  Conceptual  Model  -  The  VV&A  Team  supports  the  Federation 
Development  Team  in  developing  the  Federation  Scenarios,  Federation  Conceptual  Model, 
and  Federation  Requirements.  They  also  contribute  to  verifying  these  products  and  are  responsible  for 
validating  the  Federation  Conceptual  Model. 

•  Phase  3:  Verify  Federation  Design  -  The  VV&A  Team  supports  selecting  the  federates  and  preparing 
the  Federation  Design.  They  also  contribute  to  verifying  the  Federation  Design  and  provide  an 
updated  Federation  Accreditation  Plan  and  V&V  Plan  for  inclusion  into  the  Federation  Development 
and  Execution  Plan. 

•  Phase  4:  Verify  Federation  Development  Products  -  The  VV&A  Team  supports  developing  the 
FOM,  establishing  the  Federation  Agreements,  and  implementing  the  Federation  Infrastructure. 
They  also  contribute  to  verifying  the  FOM,  Federation  Agreements,  and  Federation  Infrastructure. 
They  provide  any  support  needed  to  verify  and  validate  the  federate  implementations  and  collect  the 
information  from  these  V&V  processes.  Finally,  they  verify  and  validate  the  data  sets  needed  for 
federation  execution. 

•  Phase  5:  Validate  and  Accept  Federation  -  The  VV&A  Team  supports  the  Federation  Development 
Team  in  planning  the  federation  execution,  integrating  the  federation,  and  testing  the  federation. 
They  also  contribute  to  verifying  the  integrated  federation.  They  have  primary  responsibility  for 
validating  the  results  produced  by  the  integrated  federation  and  developing  the  federation 
acceptance/accreditation  recommendations  from  the  collected  V&V  evidence. 

•  Phase  6:  V&V  Federation  Output  -  The  VV&A  Team  supports  executing  the  federation  when  needed 
and  contributes  to  verifying  the  raw  execution  and  derived  output  produced  from  these  executions. 
They  then  validate  the  federation  output  when  needed. 

•  Phase  7:  Consolidate  Federation  VV&A  Products  -  The  VV&A  Team  supports  analyzing  the 
federation  output  when  needed  and  collects  and  assembles  the  products  from  the  preceding  VV&A 
activities  into  a  consolidated  package  to  support  future  reuse. 

The  seven  phases  in  this  overlay  parallel  the  seven  steps  in  the  FEDEP.  As  in  the  FEDEP,  this  overlay 
decomposes  each  of  the  VV&A  phases  into  a  set  of  interrelated  lower-level  activities  and  supporting  information 
resources. 

7.0  VV&A  OVERLAY  MODEL:  DETAILED  VIEW 

The  VV&A  overlay  describes  a  high  level  framework  for  verifying,  validating,  and  accepting  HLA 
federations.  The  intent  of  the  VV&A  overlay  is  to  specify  a  recommended  practice  that  details  the  VV&A 
phases  that  correspond  to,  support,  and  augment  the  FEDEP  steps.  Federation  stakeholders  can  leverage  this 
overlay  to  achieve  the  specific  needs  associated  with  their  intended  use. 

Figure  A 1-2  provides  a  detailed  view  of  the  VV&A  overlay  and  its  interactions  with  the  FEDEP  at  equivalent 
levels  of  abstraction.  This  view  illustrates  the  flow  of  VV&A  activities,  information,  and  products  across  the 
seven  process  phases  identified  in  Figure  Al-1.  Figure  Al-2  depicts  the  FEDEP  steps,  the  VV&A  phases, 
and  the  VV&A  information  shared  between  these  steps  and  phases. 
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Figure  A1-2:  Detailed  View  of  the  VV&A  Overlay  to  the  FEDEP. 


The  following  subsections  describe  the  lower-level  activities  associated  with  each  of  the  seven  major  VV&A 
phases  associated  with  the  FEDEP’s  seven  steps.  Figure  Al-3  identifies  the  individual  activities  that  compose 
each  of  these  of  these  VV&A  phases.  Each  activity  description  includes  the  information  required, 
the  component  tasks,  and  the  information  produced  for  that  activity.  As  in  the  FEDEP,  this  overlay  assumes 
that  once  a  product  has  been  created,  it  will  be  available  for  all  subsequent  activities;  even  though  the  product 
may  not  be  identified  as  an  input  in  the  activity  description. 
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Figure  A1-3:  Activities  Associated  with  Each  VV&A  Overlay  Phase. 


Although  many  of  the  activities  represented  in  this  overlay  diagram  appear  highly  sequential,  the  intention  is  not 
to  suggest  a  strict  waterfall  approach  to  W&A.  Rather,  this  process  illustration  is  simply  intended  to  highlight 
the  major  VY&A  activities  that  occur  during  federation  development  and  execution  and  approximately  when 
such  activities  are  first  initiated  relative  to  other  VV&A  or  FEDEP  activities.  The  activities  described  in  this 
recommended  practice  are  intended  to  be  tailored  to  meet  the  needs  of  each  individual  application.  The  guidance 
provided  in  this  recommended  practice  should  be  used  as  a  starting  point  for  developing  the  specific  approach  to 
VV&A  associated  with  federation  development  and  execution  for  the  intended  use. 


7.1  Phase  1  -  Verify  Federation  Objectives 

The  purpose  of  Phase  1  of  the  W&A  overlay  is  to  define  the  scope  of  the  VV&A  effort  and  establish  a  stable 
foundation  for  establishing  a  federation’s  validity.  This  requires  the  W&A  Team  to  understand  the  Federation 
Objectives,  the  risk  that  the  User/Sponsor  can  tolerate  in  their  intended  use  of  the  federation’s  output,  the  nature 
of  the  Federation  Referent,  the  information  needed  to  define  realistic  and  observable  Federation  Acceptability 
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Criteria,  and  the  resources  required  to  support  the  subsequent  VV&A  activities.  This  phase  should  result  in  a  set 
of  verified  Federation  Objectives  and  a  Federation  Referent  and  Federation  Acceptability  Criteria  that 
adequately  represent  those  objectives. 

Figure  Al-4  illustrates  the  key  activities  in  this  phase  of  the  overlay.  These  activities  support  the  Federation 
Development  Team  in  identifying,  clearly  describing,  and  documenting  the  problem  that  the  federation 
addresses.  Understanding  what  the  User/Sponsor  really  needs  the  federation  to  accomplish  is  essential  to  the 
development  and  VV&A  of  the  federation.  A  clear,  consistent,  and  complete  yet  concise  User/Sponsor  Needs 
statement  will  aid  the  VV&A  Team  in  understanding  and  assessing  the  federation  objectives,  requirements, 
development  plans,  and  other  products  resulting  from  exercising  the  FEDEP.  For  example,  the  User/Sponsor 
Needs  statement  should  include  such  information  as  high  level  descriptions  of  the  critical  systems  of  interest, 
initial  estimates  of  fidelity  requirements,  key  scenario  events,  and  output  data  requirements.  These  insights  are 
important  for  early  VV&A  planning. 


Figure  Al-4:  Verify  Federation  Objectives  (Phase  1)  Activity  Diagram. 


7.1.1  Activity  1.1  -  Support  Identifying  User/Sponsor  Needs 

This  activity  assists  the  Federation  Development  Team  in  identifying,  clearly  describing,  and  documenting  the 
problem  that  the  federation  addresses.  Understanding  what  the  User/Sponsor  really  wants  the  federation  to 
accomplish  is  essential  to  the  development  and  VV&A  of  the  federation.  A  clear,  consistent,  and  complete  yet 
concise  User/Sponsor  Needs  statement  will  aid  the  VV&A  Team  in  understanding  and  assessing  the 
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federation  objectives,  requirements,  development  plans,  and  other  products  resulting  from  exercising  the 
FEDEP.  For  example,  the  User/Sponsor  Needs  statement  should  include  such  information  as  high  level 
descriptions  of  the  critical  systems  of  interest,  initial  estimates  of  fidelity  requirements,  key  scenario  events, 
and  output  data  requirements.  These  insights  are  important  for  early  VV&A  planning. 

The  Federation  Development  Team  should  ensure  that  the  user’s  representational  needs  (i.e.,  what  the 
federation  needs  to  represent  and  how  correct  those  representations  need  to  be)  are  captured  and  understood  in 
the  course  of  this  activity.  In  addition,  the  impact  of  federation  use  should  be  assessed  and  the  User/Sponsor 
tolerances  to  risk  should  be  estimated.  As  part  of  this  assessment,  the  VV&A  Team  should  identify,  quantify 
using  consistent  units,  and  rank  the  User/Sponsor  perceptions  of  the  impact  of  the  federation  producing 
incorrect  results  The  VV&A  Team  should  assemble  the  information  on  the  impacts  and  risks  of  using  a 
federation  into  the  Federation  Use  Impact  Assessment. 

The  process  of  identifying  User/Sponsor  Needs  will  provide  insight  into  how  correct  the  federation  needs  to  be 
and  how  much  detail  the  referent  needs  to  provide.  This  information  will  help  to  identify  the  Federation 
Referent.  The  VV&A  Team  should  also  assess  the  credibility  of  any  existing  domain  descriptions  recommended 
by  the  User/Sponsor. 

Finally,  as  part  of  the  corresponding  FEDEP  activity  the  Federation  Development  Team  begins  to  identify  the 
resources  that  will  be  available  to  support  the  federation  (e.g.,  personnel,  tools,  and  facilities)  as  well  as  any 
known  constraints  that  may  affect  how  the  federation  is  developed  (e.g.,  required  federation  participants, 
due  dates,  site  and  federation  management  requirements,  and  security  requirements).  The  VV&A  Team  can 
supply  information  to  support  this  task.  For  example,  they  might  have  input  to  the  selection  of  particular  tools 
or  facilities  that  would  benefit  both  development  and  V&V  activities. 

7. 1.1.1  Information  Required 

•  Overall  plans  (from  the  User/Sponsor’s  perspective) 

•  Existing  domain  descriptions 

•  Information  on  available  resources 

•  User/Sponsor  needs  input  including  program  objectives 

7. 1.1. 2  Functions  (Tasks) 

•  Support  analysis  of  the  program  objectives  to  identify  the  specific  purpose  and  objective(s)  that  motivate 
development  and  execution  of  a  federation 

•  Support  identifying  the  available  resources  and  known  development  and  execution  constraints, 
as  appropriate 

•  Assess  the  User/Sponsor’s  belief  in  the  credibility  of  the  existing  domain  descriptions 

•  Identify,  quantify  using  consistent  units,  where  possible,  and  rank  the  User/Sponsor  perceptions  of  the 
impact  of  using  the  federation 

•  Determine  the  User/Sponsor  tolerances  for  risk  of  incurring  the  impacts  of  the  federation  producing 
incorrect  results  for  their  intended  uses 


RTO-TR-MSG-019 


A1  - 15 


APPENDIX  1  -  VV&A  OVERLAY  TO  THE  FEDEP 


ORGANIZATION 


•  Assemble  the  information  on  the  impacts  and  risks  of  using  a  federation  into  the  Federation  Use  Impact 
Assessment 

•  Support  documentation  of  the  User/Sponsor  Needs 

7. 1.1. 3  Information  Produced 

•  Input  to  the  detailed  User/Sponsor  Needs  (e.g.,  analysis  results,  revision  suggestions) 

•  Input  to  the  available  resources  and  known  development  and  execution  constraints 

•  Credibility  of  the  existing  domain  descriptions 

•  Federation  Use  Impact  Assessment 

7.1.2  Activity  1.2  -  Plan  Accreditation  Activities 

This  activity  focuses  on  planning  the  federation  accreditation  activities.  The  resulting  plan  identifies  the 
information  required  to  support  an  acceptance  decision  and  the  tasks  needed  to  develop  that  information. 
The  Federation  Accreditation  Plan  is  necessary  for  planning  the  V&V  activities,  guiding  the  accreditation 
process,  and  building  the  Federation  Development  and  Execution  Plan. 

An  acceptance  decision  (i.e.,  a  decision  to  use  a  federation’s  results  to  serve  an  intended  use)  precedes 
federation  use.  A  formal  accreditation  decision  follows  when  required.  The  accreditation  activities,  and  the 
planning  for  them,  support  either  formal  accreditation  or  informal  acceptance. 

This  activity  analyzes  the  User/Sponsor  Needs  and  the  Federation  Use  Impact  Assessment,  as  well  as  other 
information,  to  determine  the  needed  accreditation  tasks  then  organizes  those  tasks  into  an  executable 
sequence.  For  example,  the  accreditation  tasks  might  include  formulating  the  acceptability  criteria, 
constructing  the  referent,  performing  a  risk  assessment,  and  developing  the  federation  acceptance/ 
accreditation  recommendations.  The  Federation  Accreditation  Plan  also  identifies  the  information  that  the 
V&V  activities  should  produce.  Finally,  the  accreditation  tasks  are  arranged  into  a  schedule  and  estimates  of 
the  resources  needed  to  perform  the  accreditation  are  made. 

This  activity  assumes  that  the  User/Sponsor  understands  and  has  expressed  their  perspective  on  what  they 
need  for  the  federation  to  do,  how  correct  it  needs  to  be,  and  the  process  they  intend  to  use  to  identify  and 
assign  resources  to  the  accreditation  and  V&V  processes.  It  also  assumes  that  the  User/Sponsor  understands 
the  impact  and  risk  of  applying  the  federation  to  their  intended  use. 

7. 1.2.1  Information  Required 

•  User/Sponsor  Needs 

•  Overall  plans  (from  the  User/Sponsor’s  perspective) 

•  Federation  Use  Impact  Assessment 

•  Available  resources  and  known  development  and  execution  constraints 

•  User/Sponsor  input  on  the  credibility  expected  of  the  federation  results 
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7. 1.2.2  Functions  (Tasks) 

•  Derive  and  prioritize  the  acceptance/accreditation  objectives  considering  the  User/Sponsor  credibility 
expectations,  overall  User/Sponsor’s  plans,  User/Sponsor  Needs,  known  resource  constraints,  and 
Federation  Use  Impact  Assessment 

•  Identify  the  specific  tasks  needed  to  achieve  each  acceptance/accreditation  objective 

•  Determine  the  information  dependencies  between  each  acceptance/accreditation  tasks  and  organize  the 
execution  of  these  tasks  based  upon  those  dependencies 

•  Identify  the  specific  V&V  information  needed  to  support  the  federation’s  acceptance/accreditation 

•  Determine  the  dependencies  between  the  accreditation  information  needs  and  organize  those  needs 
accordingly 

•  Devise  a  preliminary  schedule  within  which  to  execute  the  acceptance/accreditation  tasks  in  the  proper 
order 

•  Estimate  the  resources  (e.g.,  time,  funding,  personnel,  tools,  federation  development  products,  information) 
required  to  perform  each  V&V  task 

•  Document  an  initial  version  of  the  Federation  Accreditation  Plan 

7. 1.2. 3  Information  Produced 

•  Initial  Federation  Accreditation  Plan 


7.1.3  Activity  1.3  -  Support  Developing  Federation  Objectives 

This  activity  supports  developing  the  Federation  Objectives.  The  Federation  Objectives  serve  as  a  foundation 
for  generating  Federation  Requirements  (i.e.,  translating  high-level  User/Sponsor  expectations  into  more 
concrete,  measurable  federation  goals). 

This  activity  requires  close  collaboration  between  the  federation  User/Sponsor  and  the  Federation 
Development  Team,  including  the  VV&A  Team,  to  ensure  that  the  original  needs  are  properly  analyzed  and 
correctly  interpreted  and  that  the  resulting  objectives  completely  capture  the  nuances  of  the  intended  use. 

The  VV&A  Team  actively  participates  in  the  FEDEP  tasks  of: 

•  Analyzing  sponsor  needs  with  an  emphasis  upon  those  needs  that  have  implications 
federation’s  representations 

•  Assessing  federation  feasibility  and  risk  with  the  insight  gained  from  the  Federation  Use 
Assessment 

•  Contributing  to  documenting  Federation  Objectives 

•  Defining  and  documenting  the  initial  Federation  Development  and  Execution  Plan 

•  Identifying  the  initial  tools  to  support  federation  development  that  also  support  the  VV&A  activities 

•  Supporting  the  User/Sponsor  review  of  the  Federation  Objectives  and  reconciling  differences 


for  the 
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In  addition,  the  VV&A  Team  should  determine  their  impact  upon  the  test,  configuration  management, 
security,  and  quality  assurance  plans,  and  provide  input  to  those  plans  to  reflect  those  impacts.  For  example, 
the  V&V  activities  should  collaborate  closely  with  the  testing  efforts  in  order  to  leverage  the  testing  activities 
for  results  validation.  The  VV&A  Team  should  work  with  the  testing  team  to  ensure  that  the  federation  testing 
will  generate  the  data  necessary  to  support  V&V  activities.  Similar  cooperation  is  needed  between  the  VV&A 
efforts  and  those  for  configuration  management  and  quality  assurance.  Measures  to  assure  the  security  of  the 
federation  and  its  information  can  impact  the  representational  capabilities  of  the  federation.  The  VV&A  Team 
should  review  the  security  plans  to  ensure  that  the  Federation  Development  Team  is  aware  of  any 
representational  impacts  of  the  planned  security  measures. 

7. 1.3.1  Information  Required 

•  User/Sponsor  Needs 

•  Existing  domain  descriptions 

•  Federation  Use  Impact  Assessment 

•  Initial  Federation  Accreditation  Plan 

•  Initial  plans  from  the  other  Federation  Development  Team  members  (e.g.,  security,  configuration 
management,  testing) 

•  Information  on  the  resources  available  for  the  federation  VV&A  activities 

7. 1.3. 2  Functions  (Tasks) 

•  Support  analyzing  the  User/Sponsor  Needs  and  deriving  the  Federation  Objectives  from  those  needs, 
particularly  the  representational  objectives 

•  Support  assessing  the  federation  feasibility  and  risk,  particularly  the  risk  associated  with  the  federation’s 
use 

•  Support  defining,  prioritizing,  and  documenting  the  Federation  Objectives 

•  Support  defining  and  documenting  the  initial  Federation  Development  and  Execution  Plan,  particularly 
those  aspects  involving,  affecting  or  affected  by  the  VV&A  activities 

•  Support  identifying  the  potential  tools  to  support  federation  design,  development,  and  execution  that  also 
support  the  VV&A  activities 

•  Determine  and  generate  input  to  the  test,  configuration  management,  security,  and  quality  assurance 
planning  documents 

•  Support  the  User/Sponsor  review  of  the  Federation  Objectives  and  reconciling  differences 

7. 1.3. 3  Information  Produced 

•  Input  to  the  Federation  Objectives 

•  Input  to  the  federation  risk  assessment 

•  Input  to  the  initial  Federation  Development  and  Execution  Plan 

•  Input  to  the  test,  configuration  management,  security,  and  quality  assurance  planning  documents 

•  Input  to  the  initial  federation  design,  development,  and  execution  tool  selection 
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7.1.4  Activity  1.4  -  Contribute  to  Verifying  Federation  Objectives 

This  activity  determines  that  the  Federation  Objectives  correctly  and  adequately  represent  the  User/Sponsor 
Needs  and  may  indicate  the  need  for  corrections  or  additions.  An  incomplete  statement  of  Federation 
Objectives  impacts  the  ability  to  meet  the  User/Sponsor  Needs  while  overly  broad  Federation  Objectives 
could  unnecessarily  tax  the  available  resources. 

Verifying  the  completeness  and  consistency  of  the  Federation  Objectives,  addressing  both  their  content  and 
form,  is  essential  to  ensure  that  they  provide  a  clear,  solid,  and  unmistakable  foundation  for  further  federation 
development  and  use.  Further,  documenting  the  traceability  between  the  Federation  Objectives  and  the  User/ 
Sponsor  Needs  is  essential  to  diagnosing  any  problems  and  making  design  and  development  tradeoff 
decisions. 

The  VV&A  Team  should  approach  verifying  the  Federation  Objectives  as  an  integral  part  of  the  Federation 
Development  Team.  The  Federation  Development  Team  members  can  successfully  share  the  burden  of 
verification  in  many  different  ways  depending  upon  the  specifics  of  the  organizations  involved,  the  distribution 
of  responsibilities  within  that  team,  and  the  nature  of  the  Federation  Objectives. 

Regardless  of  who  performs  specific  verification  tasks,  the  tasks  in  this  activity  assume  that  the  VV&A  Team 
has  complete  access  to  the  results  from  all  verification  tasks  as  well  as  the  Federation  Objectives  and  the 
User/Sponsor  Needs  and  can  get  clarifications  from  the  User/Sponsor  upon  request. 

7. 1.4.1  Information  Required 

•  User/Sponsor  Needs 

•  Federation  Objectives 

7. 1.4. 2  Functions  (Tasks) 

•  Check  the  Federation  Objectives  for  internal  consistency  and  completeness 

•  Verify  the  Federation  Objectives  against  the  User/Sponsor  Needs  and  provide  feedback  on  the  results  of 
that  consistency  check 

•  Contribute  to  documenting  the  Federation  Objectives  verification  results 

•  Contribute  to  documenting  the  traceability  between  the  Federation  Objectives  and  the  User/Sponsor 
Needs 

7. 1.4. 3  Information  Produced 

•  Federation  Objectives  verification  results 

•  Information  describing  the  traceability  between  the  Federation  Objectives  and  the  User/Sponsor  Needs 

7.1.5  Activity  1.5  -  Assemble  Federation  Referent 

This  activity  assembles  the  referent  that  serves  as  the  basis  for  assessing  the  correctness  of  the  federation’s 
representation.  Referent  knowledge  can  come  from  direct  observations  of  the  represented  phenomena,  validated 
theory  describing  the  phenomena,  validated  simulations  representing  the  phenomena,  subject  matter  expert 
knowledge  of  the  phenomena,  and  combinations  of  information  from  these  different  sources.  In  addition,  it  may 
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come  from  several  independent  sources  of  the  same  type.  The  content  of  the  Federation  Referent  depends  upon 
the  intended  use  of  the  federation  and  the  contributing  federate  representations.  Further,  the  Federation  Referent 
need  only  cover  those  areas  where  one  needs  to  judge  the  representation’s  correctness.  The  Federation 
Objectives  can  provide  the  information  needed  to  tailor  the  referent’s  scope. 

The  tasks  in  this  activity  represent  the  effort  required  to: 

*  Collect  the  information  from  the  different  referent  sources 

*  Combine  that  information  into  a  single  consistent  referent  for  the  entire  federation’s  representation 

Collecting  the  referent  information  includes  defining  the  referent’s  scope  of  coverage,  identifying  credible 
sources,  and  acquiring  the  information  from  which  the  referent  will  be  constructed.  Information  describing  the 
same  phenomena  may  need  to  be  combined  into  a  consistent  description.  That  process  involves  selecting  the 
combination  or  fusion  techniques,  actually  exercising  those  techniques  on  the  contributing  information,  and 
estimating  the  uncertainties  associated  with  the  integrated  referent. 

Federation  Referents  may  be  assembled  from  the  referents  used  for  the  individual  federates  or  some 
combination  of  information  from  federate  referents  and  from  external  sources,  then  tailored  to  suit  the  scope 
defined  by  the  Federation  Objectives.  When  collecting  referent  information,  descriptions  of  the  uncertainties 
associated  with  that  information  should  be  collected  at  the  same  time  where  available.  If  not  available  then 
those  uncertainties  should  be  estimated  where  possible. 

Above  all,  the  Federation  Referent  should  be  credible  to  the  User/Sponsor  because  they  should  be  confident 
that  the  federation  has  sufficient  correctness  to  serve  their  intended  use.  This  constraint  implies  that  the 
User/Sponsor  should  trust  the  credibility  of  the  referent  information  sources,  the  techniques  used  to  fuse  that 
information  into  a  single  referent,  and  the  people  applying  those  fusion  techniques.  Often  the  User/Sponsor 
will  have  a  specific  referent  source  in  mind  (e.g.,  existing  domain  descriptions,  training  documents,  legacy 
simulations).  The  VV&A  Team  needs  to  recognize  these  preferences  and  gain  access  to  these  sources  to 
support  Federation  Referent  construction. 

7. 1.5.1  Information  Required 

•  Verified  Federation  Objectives 

•  User/Sponsor  Needs 

•  User/sponsor  suggestions  for  credible  referent  sources 

•  Federation  Use  Impact  Assessment 

•  Initial  Federation  Accreditation  Plan 

•  Existing  domain  descriptions 

•  Referents  used  for  past  federations  with  similar  objectives 

7. 1.5. 2  Functions  (Tasks) 

•  Identify  the  referent  needs  from  the  Federation  Objectives 

•  Select  credible  referent  information  sources 

•  Acquire  referent  information  from  the  selected  sources 
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•  Choose  referent  information  fusion  techniques  where  needed 

•  Combine  referent  information  from  multiple  sources  into  a  coherent  referent 

•  Estimate  the  error  characteristics  of  the  integrated  Federation  Referent,  where  possible 

•  Estimate  the  uncertainties  associated  with  the  errors  in  the  Federation  Referent,  where  possible 

•  Document  the  Federation  Referent  description 

7. 1.5. 3  Information  Produced 

•  Federation  Referent. 

7.1.6  Activity  1.6  -  Define  Federation  Acceptability  Criteria 

Defining  the  Federation  Acceptability  Criteria  should  clearly  and  unmistakably  outline  the  scope  of  the 
VV&A  effort  and  establish  a  stable  foundation  for  assessing  a  federation’s  validity.  Federation  Acceptability 
Criteria  are  derived  from  the  Federation  Objectives  and  are  measurable  criteria  that  explicitly  define  the  limits 
of  federation  validity. 

The  Federation  Acceptability  Criteria  should  be  necessary  and  sufficient  to  assess  the  federation’s  validity  and 
acceptability  for  the  intended  use.  They  should  be  chosen  so  that  passing  all  of  them  implies  fitness  for  the 
intended  use  and  failing  any  one  of  them  requires  the  placing  of  limitations  on  the  original  intended  use. 
While  a  general  set  of  Federation  Acceptability  Criteria  addresses  a  wide  range  of  federation  capabilities, 
those  of  primary  interest  to  the  VV&A  team  define  the  representational  aspects  of  the  federation,  including: 

•  Output  data  (including  visual  and  audio  presentations)  that  the  federation  should  generate 

•  Input  data  that  the  federation  needs  to  accept,  the  ranges  of  values  of  those  inputs  over  which  the 
federation  needs  to  generate  valid  output  and  the  errors  in  that  input  (defined  against  the  referent)  that 
the  federation  needs  to  tolerate  while  still  generating  valid  output 

•  Entities,  the  properties  that  represent  the  state  of  those  entities,  and  the  relationships  between  those 
entity  properties  that  the  federation  needs  to  represent  to  generate  valid  output  when  provided  sufficient 
input  data 

•  Required  parameters  that  can  be  controlled  and  varied  during  federation  execution,  including  man- 
machine-  interaction 

•  Minimum  confidence  required  that  each  acceptability  criterion  has  been  met 

This  activity  assumes  that  the  User/Sponsor  or  their  representative  reviews  the  Federation  Acceptability 
Criteria  for  completeness  and  correctness.  The  VV&A  Team  will  likely  define  the  Federation  Acceptability 
Criteria  iteratively,  beginning  with  preliminary  criteria  then  refining  these  as  they  gain  more  information 
through  the  federation  development  process.  This  activity  produces  an  initial  set  of  criteria  that  later  activities 
will  refine  and  extend. 

7. 1.6.1  Information  Required 

•  Verified  Federation  Objectives 

•  Federation  Referent 

•  Initial  Federation  Accreditation  Plan 
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•  User/Sponsor  Needs 

•  Federation  Use  Impact  Assessment 

•  Relevant  measures  of  effectiveness  and  measures  of  performance  related  to  the  required  representational 
capabilities 

7. 1.6. 2  Functions  (Tasks) 

•  Derive  and  verify  appropriate  acceptability  criteria  from  the  User/Sponsor  Needs,  the  Federation  Objectives, 
and  the  Federation  Referent 

•  Verify  the  internal  consistency  and  completeness  of  the  resulting  acceptability  criteria  set 

•  Verify  the  completeness  and  correctness  of  the  Federation  Acceptability  Criteria  with  the  User/Sponsor 
Needs  and  Federation  Objectives 

•  Verify  the  consistency  of  the  acceptability  criteria  with  the  User/Sponsor’s  intent 

•  Contribute  to  documenting  the  traceability  between  the  Federation  Acceptability  Criteria  and  the  Federation 
Objectives 

•  Document  the  Federation  Acceptability  Criteria 

7. 1.6. 3  Information  Produced 

•  Initial  Federation  Acceptability  Criteria 

•  Information  describing  the  traceability  between  the  Federation  Acceptability  Criteria  and  the  Federation 
Objectives 

7.1.7  Activity  1.7  -  Plan  V&V  Activities 

This  activity  plans  the  V&V  effort  to  support  the  federation  acceptance/accreditation  recommendations. 
The  Federation  Acceptability  Criteria  scope  this  plan.  Failure  to  carefully  review  the  information  to  support 
this  planning  will  place  the  successful  completion  of  the  V&V  process  at  risk. 

This  activity  involves  identifying  the  V&V  objectives  from  the  accreditation  information  needs  defined  in  the 
Federation  Accreditation  Plan  and  the  Federation  Acceptability  Criteria,  prioritizing  those  objectives,  placing 
them  in  perspective  with  the  User/Sponsor  Needs  (including  the  correctness  required  for  the  intended  use), 
determining  the  specific  tasks  needed  to  achieve  those  objectives,  and  assigning  resources  to  complete  those 
tasks.  Examples  of  these  tasks  include  validating  the  conceptual  model,  verifying  the  Federation  Object 
Model,  and  validating  the  federation  results. 

The  VV&A  Team  needs  to  understand  the  Federation  Objectives,  the  risk  that  the  User/Sponsor  can  tolerate  in 
their  intended  use  of  the  federation’s  output,  the  information  that  the  acceptance/accreditation  process  needs 
from  the  V&V  process,  and  the  planned  resources  available  to  support  the  V&V  activities.  If  the  VV&A  Team 
believes  more  resources  are  required  to  perform  the  tasks  then  they  should  make  the  Federation  Manager  or 
User/Sponsor  aware  of  that  need. 

This  activity  will  produce  an  initial  V&V  plan  that  can  then  be  refined  as  the  VV&A  Team  gains  information 
through  the  federation  development  process. 
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7. 1.7.1  Information  Required 

•  Initial  Federation  Accreditation  Plan 

•  Initial  Federation  Acceptability  Criteria 

•  Verified  Federation  Objectives 

•  User/Sponsor  Needs 

•  User/Sponsor  input  as  to  the  credibility  expected  of  the  federation  results 

•  Overall  plans  (from  the  User/Sponsor’s  perspective) 

•  Initial  Federation  Development  and  Execution  Plan 

•  Initial  plans  from  the  other  Federation  Development  Team  members  (e.g.,  security,  configuration 
management,  testing) 

•  Federation  Use  Impact  Assessment 

•  Available  resources  and  known  development  and  execution  constraints 

7. 1. 7.2  Functions  (T asks) 

•  Derive  and  prioritize  V&V  objectives  from  the  acceptance/accreditation  information  needs,  known 
resource  constraints,  Federation  Use  Impact  Assessment,  Federation  Accreditation  Plan,  and  other  FEDEP 
planning  documents 

•  Identify  the  specific  tasks  needed  to  achieve  each  V&V  objective  based  on  the  most  appropriate  and  cost- 
effective  methods,  tools,  and  techniques 

•  Determine  the  information  dependencies  between  each  V&V  task  and  organize  the  execution  of  these 
tasks  based  upon  those  dependencies 

•  Devise  a  preliminary  schedule  within  which  to  execute  the  V&V  tasks  in  the  proper  order 

•  Estimate  the  resources  (e.g.,  time,  funding,  personnel,  tools,  federation  development  products,  information) 
required  to  perform  each  V&V  task 

•  Identify  candidate  V&V  tools 

•  Document  the  initial  Federation  V&V  Plan  based  on  the  results  of  tasks  above 

7. 1.7. 3  Information  Produced 

•  Initial  Federation  V&V  Plan 

7.2  Phase  2  -  Verify  and  Validate  Federation  Conceptual  Model 

The  primary  purpose  of  Phase  2  of  the  VV&A  overlay  is  to  verify  and  validate  the  Federation  Conceptual 
Model.  The  VV&A  Team  also  supports  developing  and  contributes  to  verifying  the  Federation  Scenarios, 
Federation  Conceptual  Model,  and  Federation  Requirements.  These  activities  contribute  to  ensuring  that  a 
federation  built  to  meet  the  Federation  Requirements  can  fully  realize  the  representational  capabilities 
described  in  the  validated  Federation  Conceptual  Model  and  enables  it  to  execute  the  Federation  Scenarios. 
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Development  and  verification  of  the  Federation  Scenarios  and  Federation  Conceptual  Model  may  be  done  in 
parallel  and  iteratively.  Once  this  iteration  process  has  produced  a  stable  Federation  Conceptual  Model, 
the  VV&A  Team  can  validate  it. 

Figure  Al-5  illustrates  the  key  activities  in  this  phase  of  the  VV&A  overlay.  The  subsections  that  follow 
describe  each  of  these  activities. 


Figure  Al-5:  V&V  Federation  Conceptual  Model  (Phase  2)  Activity  Diagram. 


7.2.1  Activity  2.1  -  Support  Developing  Federation  Scenarios 

This  activity  supports  producing  a  functional  specification  of  the  envisioned  federation  scenario.  A  Federation 
Scenario  serves  as  a  bounding  mechanism  on  the  scope  of  conceptual  modeling  activities.  At  a  minimum, 
the  Federation  Scenario  defines  the  initial  and  termination  conditions  for  the  federation  execution  in  terms  of 
the  federation’s  representations. 
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A  Federation  Scenario  should  describe  the  types  and  numbers  of  major  entities  that  should  be  represented  in  a 
federation;  functional  descriptions  of  their  capabilities,  behaviors,  and  relationships  over  time,  and  the 
relevant  environmental  conditions  that  influence  the  simulated  entities. 

The  VV&A  Team  should  ensure  that  the  Federation  Scenarios  adequately  meet  the  verified  Federation 
Objectives  and  are  credible  to  the  User/Sponsor.  To  do  this  the  VV&A  Team  should  determine  whether  the 
proposed  scenario  development  standards,  tools,  and  techniques  are  suitable  to  support  scenario  verification 
activities.  Related  tasks  for  the  VV&A  Team  would  be  to  locate  and  collect  V&V  histories  of  the  candidate 
scenarios,  as  well  as  to  locate  and  collect  information  describing  the  pedigree  of  the  existing  domain 
descriptions.  The  VV&A  supports  the  Federation  Development  Team  in  vignette  definition. 

This  activity  assumes  that  the  W&A  Team  has  adequate  access  to  the  information  supporting  Federation 
Scenario  development  (e.g.,  authoritative  data  sources)  to  assess  scenario  credibility.  This  activity  also  assumes 
iterative  interaction  between  the  development  of  the  Federation  Scenario  and  the  Federation  Conceptual  Model. 

7. 2. 1.1  Information  Required 

•  Verified  Federation  Objectives 

•  Existing  scenarios 

•  Federation  Conceptual  Model 

•  Existing  domain  descriptions 

7. 2. 1.2  Functions  (Tasks) 

•  Determine  the  suitability  of  the  planned  scenario  development  standards,  tools,  and  techniques  to  support 
scenario  validation  activities 

•  Locate  and  collect  V&V  histories  of  the  candidate  scenarios 

•  Evaluate  the  appropriateness  of  the  existing  scenario  resources  for  achieving  the  Federation  Objectives 

•  Locate  and  collect  information  describing  the  pedigree  of  the  existing  domain  descriptions 

•  Contribute  to  validating  the  Federation  Scenarios  against  the  Federation  Acceptability  Criteria  and  the 
Federation  Referent,  as  needed 

•  Evaluate  the  credibility  and  applicability  of  the  domain  descriptions  for  achieving  the  Federation  Objectives 

•  Support  defining  one  or  more  representative  vignettes  of  federation  events  that,  once  executed, 
will  produce  the  data  necessary  to  achieve  Federation  Objectives 

7. 2. 1.3  Information  Produced 

•  Input  to  the  Federation  Scenarios 

7.2.2  Activity  2.2  -  Contribute  to  Verifying  Federation  Scenarios 

The  primary  goals  for  verifying  the  Federation  Scenarios  are  to  demonstrate  that  they  fully  cover  the  verified 
Federation  Objectives  and,  if  required,  to  indicate  the  need  for  any  corrective  actions.  The  Federation 
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Scenarios  outline  the  simulation  executions  and  if  they  do  not  fully  cover  the  intended  use  then  achieving  the 
Federation  Objectives  could  be  endangered. 

This  activity  should  verily  the  internal  consistency  and  completeness  of  the  Federation  Scenarios  ensure  that 
they  are  clearly,  completely,  and  correctly  documented,  and  make  certain  that  they  are  traceable  back  to  the 
Federation  Objectives.  Like  the  Federation  Objectives,  documenting  the  traceability  of  the  Federation 
Scenarios  links  them  to  User/Sponsor  Needs  through  the  Federation  Objectives.  This  traceability  contributes 
to  diagnosing  any  problems  and  making  design  and  development  tradeoff  decisions. 

Also  as  with  the  Federation  Objectives,  the  VV&A  Team  should  approach  verifying  the  Federation  Scenarios 
as  an  integral  part  of  the  Federation  Development  Team  and  expect  to  share  the  burden  of  verification  in  one 
of  many  different  ways  depending  upon  the  specifics  of  the  organizations  involved,  the  distribution  of 
responsibilities  within  that  team,  and  the  nature  of  the  Federation  Objectives. 

Regardless  of  who  perfonns  specific  verification  tasks,  this  activity  assumes  that  the  VV&A  Team  has 
complete  access  to  the  results  from  all  verification  tasks  as  well  as  the  Federation  Scenarios.  This  activity  also 
assumes  that  Federation  Scenarios  and  Federation  Conceptual  Model  are  developed  in  concert  over  several 
iterations.  Verification  of  the  Federation  Scenarios  may  also  occur  iteratively. 

7. 2. 2.1  Information  Required 

•  Verified  Federation  Objectives 

•  Federation  Scenarios 

•  Federation  V&V  Plan 

7. 2.2.2  Functions  (Tasks) 

•  Verify  the  internal  consistency  and  completeness  of  the  Federation  Scenarios 

•  Verify  the  consistency  of  the  Federation  Scenarios  with  the  Federation  Objectives 

•  Contribute  to  documenting  the  Federation  Scenarios  verification  results 

•  Contribute  to  documenting  the  traceability  between  the  Federation  Scenarios  and  the  Federation  Objectives 

7.2.23  Information  Produced 

•  Federation  Scenarios  verification  results 

•  Information  describing  the  traceability  between  the  Federation  Scenarios  and  the  Federation  Objectives 

7.2.3  Activity  2.3  -  Support  Developing  Federation  Conceptual  Model 

This  activity  supports  producing  a  Federation  Conceptual  Model.  The  Federation  Conceptual  Model  is  a 
conceptual  representation  of  the  intended  problem  space  based  upon  the  Federation  Development  Team’s 
interpretation  of  the  verified  Federation  Objectives.  A  Federation  Conceptual  Model  provides  an 
implementation-independent  representation  that  serves  as  a  vehicle  for  transforming  Federation  Objectives 
into  functional  and  behavioral  descriptions  for  system  and  software  designers.  The  model  also  provides  a 
crucial  traceability  link  between  the  Federation  Objectives  and  the  Federation  Design  implementation. 
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The  Federation  Development  Team,  with  input  from  the  VV&A  Team,  has  the  discretion  to  choose  the 
presentation  style  most  useful  for  describing  the  entities  and  actions  that  need  to  be  included  in  the  federation 
in  order  to  achieve  all  Federation  Objectives.  A  Federation  Conceptual  Model  needs  to  be  carefully  evaluated 
before  continuing  to  the  succeeding  stages  of  federation  development.  At  a  minimum,  the  User/Sponsor 
should  review  key  processes  and  events  to  ensure  the  adequacy  of  the  conceptual  representation. 

The  VV&A  Team  participates  as  part  of  the  Federation  Development  Team  from  the  initial  specification  of 
the  Federation  Conceptual  Model  through  its  evolution  during  subsequent  federation  development  steps. 
They  can  assist  Federation  Development  Team  to  determine  whether  or  not  existing  conceptual  models  or 
domain  descriptions  should  be  used  as  the  foundation  for  conceptual  model  construction  based  upon  the 
Federation  Objectives.  The  VV&A  Team  can  also  contribute  to  establishing  the  credibility  of  the  Federation 
Conceptual  Model  by  locating  and  collecting  information  describing  the  pedigrees  of  the  domain  descriptions 
and  by  explicitly  identifying  the  underlying  assumptions  and  limitations  of  the  Federation  Conceptual  Model. 

7.2.3. 1  Information  Required 

•  Verified  Federation  Objectives 

•  Existing  domain  descriptions 

•  Federation  Scenarios 

•  Existing  conceptual  models 

7. 2. 3. 2  Functions  (Tasks) 

•  Support  choosing  the  technique  and  format  for  developing  and  documenting  the  Federation  Conceptual 
Model  in  order  to  better  support  V&V  activities 

•  Evaluate  the  applicability  of  existing  conceptual  models  to  the  Federation  Objectives 

•  Locate  and  collect  information  describing  the  pedigree  of  the  authoritative  domain  descriptions  used  for 
conceptual  model  construction 

•  Evaluate  the  applicability  of  the  domain  descriptions  to  the  Federation  Objectives 

•  Contribute  to  identifying  and  documenting  the  assumptions  and  limitations  associated  with  the  Federation 
Conceptual  Model 

7. 2. 3. 3  Information  Produced 

•  Input  to  the  Federation  Conceptual  Model 

•  Input  to  documentation  of  the  assumptions  and  limitations  underlying  the  Federation  Conceptual  Model 

7.2.4  Activity  2.4  -  Contribute  to  Verifying  Federation  Conceptual  Model 

This  activity  verifies  the  internal  consistency,  completeness,  and  correctness  of  the  Federation  Conceptual 
Model  and  its  consistency  with  the  verified  Federation  Objectives  and  the  Federation  Scenarios.  This  step 
assures  that  the  representations  defined  in  the  Federation  Conceptual  Model  are  internally  sound  and  can 
support  the  execution  of  the  Federation  Scenarios.  This  creates  a  firm  foundation  for  conceptual  model 
validation  and  the  derivation  of  the  Federation  Requirements. 
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This  activity  verifies  that  the  Federation  Conceptual  Model  contains  the  content  defined  in  the  FEDEP 
(e.g.,  entity  state  and  behavior  representations)  in  the  form  specified  by  the  Federation  Development  Team. 
It  also  verifies  the  consistency  of  any  diagrams  depicting  model  structure  and  behavior,  the  correctness  of  the 
unit  transformations  in  any  mathematical  expressions,  and  the  Federation  Conceptual  Model’s  consistency 
with  the  Federation  Scenarios.  This  activity  documents  the  traceability  between  the  Federation  Conceptual 
Model  and  the  Federation  Scenarios  and  between  the  Federation  Conceptual  Model  and  the  Federation 
Objectives. 

This  activity  assumes  that  the  Federation  Scenarios  and  the  Federation  Conceptual  Model  can  be  developed  in 
concert  over  several  iterations.  Verification  of  the  Federation  Conceptual  Model  may  also  be  iterative. 

As  with  the  other  verification  activities,  the  VV&A  Team  should  approach  verifying  the  Federation 
Conceptual  Model  as  an  integral  part  of  the  Federation  Development  Team.  The  Federation  Development 
Team  members  can  successfully  share  the  burden  of  verification  in  many  different  ways  depending  upon  the 
specifics  of  the  organizations  involved,  the  distribution  of  responsibilities  within  that  team,  and  the  nature  of 
the  Federation  Conceptual  Model. 

Regardless  of  who  performs  specific  verification  tasks,  the  tasks  in  this  activity  assume  that  the  VV&A  Team 
has  complete  access  to  the  results  from  all  verification  tasks  and  can  get  clarifications  from  the  User/Sponsor 
as  needed. 

7. 2. 4.1  Information  Required 

•  Verified  Federation  Objectives 

•  Federation  Scenarios 

•  Federation  Conceptual  Model 

•  Federation  V&V  Plan 

7. 2. 4. 2  Functions  (Tasks) 

•  Verify  the  internal  consistency  and  completeness  of  the  Federation  Conceptual  Model 

•  Verify  the  consistency  of  the  Federation  Conceptual  Model  with  the  Federation  Scenarios 

•  Contribute  to  documenting  the  Federation  Conceptual  Model  verification  results 

•  Contribute  to  documenting  the  traceability  between  the  Federation  Conceptual  Model  and  the  Federation 
Objectives 

7. 2. 4. 3  Information  Produced 

•  Federation  Conceptual  Model  verification  results 

•  Information  describing  the  traceability  between  the  Federation  Conceptual  Model  and  the  Federation 
Objectives 

7.2.5  Activity  2.5  -  Validate  Federation  Conceptual  Model 

The  Federation  Conceptual  Model  provides  the  first  meaningful  insight  into  the  federation’s  validity. 
The  Federation  Conceptual  Model  defines  the  representations  upon  which  all  subsequent  activities  build. 
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All  succeeding  verification  activities  contribute  evidence  to  the  validation  of  the  federation  by  checking  the 
consistency  of  the  Federation  Development  Team’s  intermediate  products  (i.e.,  short  of  the  functioning 
integrated  federation)  against  the  validated  conceptual  model.  None  of  those  intermediate  products  can  be 
considered  valid  unless  the  Federation  Conceptual  Model  is  valid. 

At  minimum,  the  tasks  of  this  activity  assess  the  completeness  of  the  federation’s  representations,  as  defined 
in  the  verified  Federation  Conceptual  Model,  against  the  Federation  Acceptability  Criteria.  If  the  Federation 
Conceptual  Model  contains  sufficient  detail  then  this  activity  also  assesses  its  correctness  against  the 
Federation  Acceptability  Criteria.  Finally,  this  activity  should  result  in  the  User/Sponsor’s  approval  of  the 
Federation  Conceptual  Model.  This  task  improves  the  likely  relevance  of  the  resulting  federation  to  the  User/ 
Sponsor  and  contributes  to  the  process  of  building  federation  credibility  with  those  users. 

The  extent  to  which  the  VV&A  Team  can  perform  the  recommended  conceptual  model  validation  tasks 
depends  strongly  upon  the  information  contained  in  the  Federation  Objectives,  the  detail  of  the  Federation 
Acceptability  Criteria,  the  detail  in  the  Federation  Conceptual  Model,  and  the  resource  limitations  of  the 
VV&A  Team. 

This  activity  assumes  that  the  iteration  within  the  “Perform  Conceptual  Analysis”  step  of  the  FEDEP  has 
stabilized  and  produced  a  fairly  representative  conceptual  model.  Some  federation  development  efforts  may 
choose  to  perform  iterative  validation  upon  the  Federation  Conceptual  Model  while  others  may  choose  to  wait 
until  the  end.  An  iterative  approach  may  tailor  the  validation  tasks  for  each  iteration  except  at  the  end. 
This  activity  also  assumes  that  the  Federation  Conceptual  Model  has  been  verified  before  being  validated. 

7.2.5. 1  Information  Required 

•  Verified  Federation  Conceptual  Model 

•  Federation  Acceptability  Criteria 

•  Federation  Referent 

•  Federation  V&V  Plan 

7. 2. 5. 2  Functions  (Tasks) 

•  Evaluate  the  completeness  of  the  Federation  Conceptual  Model  against  the  Federation  Acceptability 
Criteria  and  Federation  Referent  and  identify  areas  of  incompleteness 

•  Estimate  the  error  characteristics  of  the  Federation  Conceptual  Model,  where  possible 

•  Evaluate  the  correctness  of  the  Federation  Conceptual  Model  against  the  Federation  Acceptability  Criteria 
and  Federation  Referent  and  identify  areas  of  incorrectness,  where  possible 

•  Document  the  Federation  Conceptual  Model  validation  results 

•  Support  reviewing  the  contents  of  the  Federation  Conceptual  Model  with  the  User/Sponsor  by  reviewing 
the  Federation  Conceptual  Model  contents  and  the  assembled  validation  evidence  with  them 

•  Get  the  approval  of  the  User/Sponsor  for  the  validated  Federation  Conceptual  Model 
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7. 2. 5. 3  Information  Produced 

•  Federation  Conceptual  Model  validation  results 

•  Within  the  context  of  the  Federation  Referent,  identification  of  the  Federation  Acceptability  Criteria 
that  the  Federation  Conceptual  Model 

•  Meets 

•  Does  not  meet 

•  Characterization  of  the  representational  errors  that  exceed  the  Federation  Acceptability  Criteria  limits, 
where  possible 

•  User/Sponsor  approval  of  the  Federation  Conceptual  Model 

7.2.6  Activity  2.6  -  Support  Developing  Federation  Requirements 

This  activity  supports  development  of  the  Federation  Requirements.  The  Federation  Requirements  are  based 
upon  the  verified  Federation  Objectives,  the  verified  Federation  Scenarios,  and  the  validated  Federation 
Conceptual  Model.  This  activity  transforms  the  Federation  Objectives  into  specific  set  of  detailed  and  testable 
Federation  Requirements  that  provide  the  implementation-level  information  needed  to  design  and  develop  the 
federation.  The  VV&A  Team  should  support  the  Federation  Development  Team  to  ensure  that  the  Federation 
Requirements  are  clear,  unique,  and  testable. 

In  the  course  of  developing  the  Federation  Requirements,  it  may  prove  necessary  to  refine  the  Federation 
Objectives.  If  so,  the  VV&A  Team  should  determine  the  validity  impacts  of  any  modifications,  revise  the 
Federation  Acceptability  Criteria  as  needed  to  maintain  consistency  with  the  revised  Federation  Objectives, 
and  gain  User/Sponsor  concurrence  of  any  such  revisions  of  the  Federation  Acceptability  Criteria. 

The  VV&A  Team  actively  participates,  as  part  of  the  Federation  Development  Team,  in  the  development  and 
documentation  of  all  Federation  Requirements.  They  also  identify  the  validity  implications  of  requirements 
for  the  execution  enviromnent,  time  management,  data  distribution  management,  execution  management, 
federation  performance,  and  security  to  ensure  consistency  between  the  resulting  Federation  Requirements 
and  the  Federation  Acceptability  Criteria.  The  VV&A  Team  should  contribute  to  deriving  a  set  of  Federation 
Test  Criteria  from  the  Federation  Requirements  that  efficiently  determine  whether  the  federation  meets  the 
Federation  Acceptability  Criteria.  They  should  also  develop  revisions  and  extensions  to  the  Federation 
Acceptability  Criteria  to  maintain  consistency  with  the  Federation  Requirements  and  Federation  Test  Criteria. 

7.2.6. 1  Information  Required 

•  Verified  Federation  Objectives 

•  Verified  and  validated  Federation  Conceptual  Model 

•  Verified  Federation  Scenarios 

•  Federation  Referent 

•  Initial  Federation  Acceptability  Criteria 
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7. 2.6.2  Functions  (Tasks) 

•  Support  defining  the  Federation  Requirements  for  the  representational  aspects  of  the  federation 
(i.e.,  those  related  to  federation  fidelity) 

•  Support  defining  the  federation  output  requirements  from  the  Federation  Acceptability  Criteria 

•  Analyze  the  requirements  associated  with  the  execution  environment,  time  management,  data  distribution 
management,  execution  management,  federation  performance,  and  security  to  determine  their  impact  on 
and  consistency  with  the  federation  representational  requirements 

•  Support  ensuring  that  the  Federation  Requirements  are  clear,  unique,  and  testable 

•  Contribute  to  deriving  a  set  of  Federation  Test  Criteria  from  the  Federation  Requirements  that  efficiently 
determine  whether  the  federation  meets  the  Federation  Acceptability  Criteria 

•  Develop  revisions  and  extensions  to  the  Federation  Acceptability  Criteria  to  maintain  consistency  with  the 
Federation  Requirements  and  Federation  Test  Criteria 

•  Support  documenting  all  federation  requirements 

7. 2. 6. 3  Information  Produced 

•  Input  to  the  Federation  Requirements 

•  Input  to  the  Federation  Test  Criteria 

•  Revised  Federation  Acceptability  Criteria 

7.2.7  Activity  2.7  -  Contribute  to  Verifying  Federation  Requirements 

This  activity  verifies  the  internal  consistency,  completeness,  and  correctness  of  the  Federation  Requirements 
both  internally  and  against  the  validated  Federation  Conceptual  Model  and  verified  Federation  Scenarios, 
and  indicates  the  need  for  any  corrective  actions  that  might  be  necessary.  This  step  ensures  that  a  federation 
built  to  meet  the  Federation  Requirements  fully  realizes  the  representational  capabilities  described  in  the 
validated  Federation  Conceptual  Model  and  enables  it  to  execute  the  Federation  Scenarios.  Inadequate 
Federation  Requirements  can  result  in  a  federation  that  cannot  meet  the  Federation  Objectives  and 
unnecessary  Federation  Requirements  needlessly  increase  the  developmental  effort. 

This  activity  includes  ensuring  that  the  Federation  Requirements  consistently,  completely,  and  correctly 
capture  all  technical  requirements  concerning  hardware  platforms,  operating  environments,  infrastructure, 
administration  tools,  and  other  components  needed  to  realize  a  fully  functioning  federation.  This  activity  also 
documents  the  traceability  between  the  Federation  Requirements  and  the  Federation  Conceptual  Model  and 
Federation  Scenarios. 

This  activity  assumes  that  the  Federation  Requirements  and  Federation  Conceptual  Model  may  be  developed 
in  concert  over  several  iterations.  Thus,  verification  of  the  Federation  Requirements  may  also  be  iterative. 

As  with  the  other  verification  activities,  the  VV&A  Team  should  approach  verifying  the  Federation 
Requirements  as  an  integral  part  of  the  Federation  Development  Team.  Regardless  of  who  verifies  the 
Federation  Requirements,  the  tasks  in  this  activity  assume  that  the  VV&A  Team  has  complete  access  to  all 
verification  results. 
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7. 2. 7. 1  Information  Required 

•  Verified  Federation  Objectives 

•  Federation  Requirements 

•  Federation  Test  Criteria 

•  Federation  Acceptability  Criteria 

•  Federation  V&V  Plan 

7. 2. 7. 2  Functions  (T asks) 

•  Verify  the  internal  consistency  and  completeness  of  the  Federation  Requirements 

•  Verify  the  consistency,  completeness,  and  correctness  of  the  Federation  Requirements  with  the  Federation 
Objectives  and  Federation  Acceptability  Criteria 

•  Verify  the  internal  consistency  and  completeness  of  the  Federation  Test  Criteria 

•  Verify  the  consistency,  completeness,  and  correctness  of  the  Federation  Test  Criteria  with  the  Federation 
Requirements 

•  Contribute  to  documenting  the  Federation  Requirements  and  Federation  Test  Criteria  verification  results 

•  Contribute  to  documenting  the  traceability  between  the  Federation  Requirements  and  the  Federation 
Conceptual  Model,  the  Federation  Scenarios,  and  the  Federation  Acceptability  Criteria 

•  Contribute  to  documenting  the  traceability  between  the  Federation  Test  Criteria  and  the  Federation 
Requirements 

7. 2. 7. 3  Information  Produced 

•  Federation  Requirements  verification  results 

•  Federation  Test  Criteria  verification  results 

•  Information  describing  the  traceability  between  the  Federation  Requirements  and  the  Federation  Objectives 
and  Federation  Acceptability  Criteria 

•  Information  describing  the  traceability  between  the  Federation  Test  Criteria  and  the  Federation 
Requirements 

7.3  Phase  3  -  Verify  Federation  Design 

The  purpose  of  Phase  3  of  the  VV&A  overlay  is  to  support  federation  design.  The  VV&A  Team  supports  all 
of  the  activities  in  Step  3  of  the  FEDEP.  They  assist  with  evaluating  the  abilities  of  the  candidate  federates  to 
meet  the  Federation  Acceptability  Criteria,  evaluate  the  effects  of  the  federation  design  decisions  upon 
federation  validity  and  update  the  Federation  Accreditation  Plan  and  V&V  Plan.  They  also  contribute  to 
verifying  the  Federation  Design. 

Figure  Al-6  illustrates  the  key  activities  in  this  phase  of  the  VV&A  overlay.  The  subsections  that  follow 
describe  each  of  these  activities. 
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Figure  A1-6:  Verify  Federation  Design  (Phase  3)  Activity  Diagram. 


7.3.1  Activity  3.1  -  Support  Selecting  Federates 

In  this  activity,  the  VV&A  Team  supports  the  Federation  Development  Team  in  determining  the  suitability  of 
individual  simulation  systems  to  become  members  of  the  federation  being  developed. 

In  order  to  select  the  most  appropriate  federates  from  the  available  candidates,  federate  selection  criteria  need 
to  be  defined  based  upon  the  validated  Federation  Conceptual  Model,  verified  Federation  Requirements, 
and  the  Federation  Acceptability  Criteria.  These  criteria  should  reflect  the  required  entities/objects  and  events 
that  the  federates  need  to  represent  to  achieve  the  Federation  Objectives.  Each  candidate  federate  should  be 
analyzed  against  these  criteria  to  determine  their  ability  to  meet  the  Federation  Acceptability  Criteria  and  the 
rationale  for  those  selected  documented.  The  VV&A  Team  should  assist  the  Federation  Development  Team 
in  defining  these  federate  selection  criteria  and  using  them  to  analyze  candidate  federates.  In  addition, 
if  an  existing  federation  is  being  considered  for  reuse  in  part  or  whole,  the  VV&A  Team  should  assist  in 
developing  and  applying  similar  criteria. 

For  each  candidate  federate  or  federation  selected,  the  VV&A  Team  needs  to  collect  any  available  V&V  and 
use  histories  and  evaluate  them  for  quality  and  relevance  to  the  Federation  Acceptability  Criteria.  The  quality 
and  utility  of  this  historical  information  available  can  vary  greatly.  The  VV&A  Team  should  evaluate  the 
information  made  available  and  determine  if  additional  V&V  information  is  needed  and  recommend  those 
information  requirements  to  the  Federation  Development  Team. 

This  activity  assumes  that  the  VV&A  Team  can  readily  access  the  V&V  histories  of  all  of  the  candidate 
federates  as  well  as  to  the  User/Sponsor  to  assess  federate  credibility. 
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7. 3. 1.1  Information  Required 

•  Federation  Acceptability  Criteria 

•  Verified  Federation  Objectives 

•  Verified  Federation  Requirements 

•  Federate  documentation  (including  Simulation  Object  Models  (SOMs)) 

•  List  of  federate  candidates 

•  Federate  candidate  V&V  and  use  histories 

7. 3. 1.2  Functions  (Tasks) 

•  Support  defining  the  criteria  for  federate  selection  from  the  Federation  Acceptability  Criteria 

•  Support  determining  if  an  existing  reusable  federation  meets  or  partially  meets  the  Federation  Requirements 

•  Collect  the  available  V&V  and  use  histories  for  each  federate  candidate 

•  Evaluate  the  quality  and  relevance  of  this  historical  information  to  the  Federation  Acceptability  Criteria 

•  Recommend  additional  V&V  information  requirements 

•  Support  analyzing  the  ability  of  each  federate  candidate  to  meet  the  Federation  Acceptability  Criteria 

•  Support  documenting  the  rationale  for  federate  selection 

7. 3. 1.3  Information  Produced 

•  Input  to  the  criteria  for  federate  selection 

•  Input  to  candidate  federate  selection  and  selection  rationale 

•  Recommendations  of  additional  V&V  infonnation  requirements 

7.3.2  Activity  3.2  -  Support  Preparing  Federation  Design 

In  this  activity,  the  VV&A  Team  supports  the  Federation  Development  Team  in  preparing  the  Federation 
Design.  At  this  point,  the  responsibility  to  represent  the  entities  and  actions  defined  in  the  validated  Federation 
Conceptual  Model  and  Federation  Scenarios  has  been  allocated  to  candidate  federates.  Analysis  of  the 
selected  federates  is  needed  to  identify  those  that  can  best  provide  the  representational  capabilities  defined  in 
the  Federation  Conceptual  Model.  As  this  analysis  depends  on  the  documentation  of  each  candidate  federate’s 
capabilities,  this  is  when  any  limitations  of  that  documentation  and  any  impacts  those  limitations  might  have 
on  the  design  of  the  federation  are  determined.  In  addition,  an  assessment  of  whether  the  set  of  selected 
federates  provides  the  required  representational  capabilities  should  be  conducted.  Federate  modifications  or 
the  development  of  new  federates  may  be  required  to  realize  the  capabilities  needed  to  meet  the  verified 
Federation  Objectives. 

When  designing  the  federation,  the  Federation  Development  Team  may  need  to  negotiate  agreements  on 
assigned  responsibilities  or  make  various  federation  design  decisions  deemed  appropriate  based  upon 
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investigations  conducted.  Such  negotiations  and  trades  may  be  required  as  technical  issues  such  as  time 
management,  federation  management,  infrastructure  design,  runtime  performance,  and  potential 
implementation  approaches  are  considered  in  the  design  process.  It  is  important  that  the  VV&A  Team  ensure 
that  these  negotiations  and  design  decisions  are  well  documented  and  analyze  them  to  determine  the  impact 
they  will  have  upon  the  overall  validity  of  the  federation’s  representations. 

The  VV&A  Team  participates  as  part  of  the  Federation  Development  Team  in  the  federation  design  effort. 
In  addition,  they  support  the  design  effort  by  identifying  the  limitations  of  the  existing  federate  candidate 
capabilities  documentation  and  assessing  the  impact  that  those  limitations  will  have  on  federation  design 
effort. 

This  activity  assumes  that  the  VV&A  Team  can  readily  access  the  sources  documenting  the  federate 
representational  capabilities  and  all  of  the  products  from  the  federation  design  effort. 

7. 3. 2.1  Information  Required 

•  Federation  Acceptability  Criteria 

•  Verified  and  validated  Federation  Conceptual  Model 

•  Verified  Federation  Scenarios 

•  Verified  Federation  Requirements 

•  Candidate  federate  documentation 

•  Federate  candidate  V&V  and  use  histories 

7. 3.2.2  Functions  (Tasks) 

•  Identify  the  limitations  of  the  existing  federate  candidate  capabilities  documentation  and  assess  the  impact 
that  those  limitations  will  have  on  federation  design  activities 

•  Support  analyzing  the  selected  federates  and  identifying  those  federates  that  best  provide  the 
representational  capabilities  defined  by  the  Federation  Requirements  and  Federation  Acceptability  Criteria 

•  Support  allocating  the  representational  capabilities  defined  by  the  Federation  Conceptual  Model  and 
Federation  Scenarios  to  the  selected  federates  and  capture  relevant  information 

•  Determine  the  Federation  Acceptability  Criteria  that  each  selected  federate  should  support  or  contribute  to 
supporting 

•  Decompose  any  requirement  or  acceptability  criterion  to  which  multiple  federates  contribute  into 
requirements  or  acceptability  criteria  specific  to  the  federate  under  consideration 

•  Contribute  to  determining  if  federate  modifications  or  new  federates  are  necessary 

7. 3. 2. 3  Information  Produced 

•  Input  to  the  Federation  Design 

•  Suggestions  for  federate  modifications  or  new  federation  development 


RTO-TR-MSG-019 


A1  -35 


APPENDIX  1  -  VV&A  OVERLAY  TO  THE  FEDEP 


ORGANIZATION 


•  Limitations  of  existing  candidate  federate  documentation 

•  Assessment  of  the  impact  that  inadequate  federate  documentation  will  have  on  federation  validity  and 
subsequent  validation  activities 

•  Allocation  of  the  Federation  Acceptability  Criteria  to  the  individual  federates 

7.3.3  Activity  3.3  -  Contribute  to  Verifying  Federation  Design 

This  activity  verifies  the  Federation  Design,  both  internally  and  against  the  validated  Federation  Conceptual 
Model,  to  establish  it  as  solid  foundation  for  further  federation  development  and,  if  required,  to  indicate  the 
need  for  corrective  actions.  An  inadequate  Federation  Design  may,  at  best,  delay  realizing  the  federation  and, 
at  worst,  lead  to  the  development  of  a  federation  that  cannot  meet  the  verified  Federation  Objectives. 

Verifying  the  consistency,  completeness,  and  correctness  of  the  Federation  Design  internally  and  against  the 
Federation  Conceptual  Model  is  an  integral  part  of  the  design  effort.  Design  verification  should  ensure  that  the 
Federation  Design  represents  all  of  the  entities,  entity  attributes,  relationships  (both  static  and  dynamic), 
behavioral  and  transformational  aspects  of  the  entities,  and  interactions  between  entities  described  in  the 
Federation  Conceptual  Model.  It  should  also  consider  the  mapping  of  those  representational  features  to  the 
federates  participating  in  the  federation.  Design  verification  may  also  analyze  the  expected  frequencies  of 
inter- federate  communication  to  ensure  that  infrastructure  loading  conditions  do  not  occur  that  can  distort  the 
federation’s  representational  capabilities  and  cause  validity  problems. 

As  with  the  other  verification  activities,  the  VV&A  Team  should  approach  verifying  the  Federation  Design  as 
an  integral  part  of  the  Federation  Development  Team  and  share  the  responsibilities  for  design  verification  as 
required  by  the  specifics  of  the  organizations  involved  and  the  delegations  of  responsibility  within  the 
Federation  Development  Team.  Regardless  of  who  performs  the  design  verification  tasks,  the  tasks  in  this 
activity  assume  that  the  VV&A  Team  has  complete  access  to  the  design  verification  results.  This  activity 
further  assumes  that  any  design  errors  found  during  verification  are  reported  to  the  Federation  Development 
Team  for  timely  and  affordable  correction. 

7. 3. 3.1  Information  Required 

•  Verified  and  validated  Federation  Conceptual  Model 

•  Federation  Design 

•  Federate  documentation 

•  Federation  Acceptability  Criteria 

•  Federation  V&V  Plan 

7. 3. 3. 2  Functions  (Tasks) 

•  Verify  the  mapping  of  the  federate  capabilities  to  the  Federation  Design 

•  Verify  the  internal  consistency  and  completeness  of  the  Federation  Design 

•  Verify  the  consistency,  completeness,  and  correctness  of  the  Federation  Design  with  the  Federation 
Conceptual  Model 
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•  Verify  that  the  Federation  Design  sufficiently  satisfies  the  federation’s  non-representational  requirements 

•  Contribute  to  documenting  the  Federation  Design  verification  results 

•  Contribute  to  documenting  the  traceability  between  the  Federation  Design  and  the  Federation  Conceptual 
Model 

7.333  Information  Produced 

•  Federation  Design  verification  results 

•  Information  describing  the  traceability  between  the  Federation  Design  and  the  Federation  Conceptual 
Model 

7.3.4  Activity  3.4  -  Support  Preparing  Federation  Development  and  Execution  Plan 

This  activity  supports  the  Federation  Development  Team  in  preparing  a  coordinated  plan  to  guide  the 
activities  for  developing,  integrating,  testing,  and  executing  the  federation.  This  plan  includes  the  plans  for 
federation  integration,  test,  and  evaluation,  configuration  management,  security,  data  collection,  management, 
analysis,  quality  assurance,  accreditation,  and  V&V  as  well  as  supporting  tools  selection  and  use. 
This  planning  activity  will  take  place  in  concert  with  the  development  of  the  federation  development  and 
testing  schedules  that  identify  detailed  tasks  and  milestones.  A  coordinated  Federation  Development  and 
Execution  Plan  is  essential  to  being  able  to  successfully  implement  a  federation  that  can  achieve  the 
Federation  Objectives.  This  integrated  planning  effort  requires  close  collaboration  between  all  of  the  members 
of  the  Federation  Development  Team  including  the  VV&A  Team. 

This  coordinated  set  of  plans  covers  areas  critical  to  successful  V&V  of  the  federation.  Therefore,  the  VV&A 
Team  should  understand  and  contribute  to  all  aspects  of  the  coordinated  plan  where  the  federation’s 
representations  may  be  impacted.  The  VV&A  Team  should  also  examine  the  data  collection,  management, 
and  analysis  plans  to  ensure  that  their  documentation  is  complete  and  conforms  with  other  existing  plans. 
In  addition,  the  VV&A  Team  needs  to  develop  a  comprehensive  results  sampling  strategy  from  the  Federation 
Acceptability  Criteria,  Federation  Conceptual  Model,  Federation  Scenarios,  and  Federation  Test  Plan, 
and  ensure  that  the  test  and  evaluation  and  data  collection  plans  adequately  support  this  strategy.  The  VV&A 
Team  should  also  revise  the  Federation  Accreditation  Plan  and  V&V  Plan  to  reflect  any  revisions  to  the 
Federation  Acceptability  Criteria  and  to  incorporate  the  results  sampling  strategy.  Finally,  the  VV&A  Team 
should  determine  that  the  tools  they  require  are  part  of  the  tool  selection  and  management  plan  as  well  as  to 
ensure  that  the  tools  being  considered  for  execution  monitoring  and  control  can  support  the  V&V  data 
collection  needs. 

73.4.1  Information  Required 

•  Initial  Federation  Development  and  Execution  Plan 

•  Initial  Federation  Accreditation  Plan  and  V&V  Plan 

•  Federation  Acceptability  Criteria 

•  Federation  Referent 

•  Verified  and  validated  Federation  Conceptual  Model 
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•  Verified  Federation  Design 

•  Verified  Federation  Scenarios 

•  Federation  Test  Plan 

•  Verified  Federation  Requirements 

•  Quality  assurance,  configuration  management,  data  collection,  management,  and  analysis  plans 

•  Initial  federation  design,  development,  and  execution  tool  selection 

7. 3. 4. 2  Functions  (Tasks) 

•  Develop  the  federation  results  sampling  strategy  from  the  Federation  Acceptability  Criteria,  Federation 
Conceptual  Model,  Federation  Scenarios,  and  Federation  Test  Plan 

•  Update  the  Federation  Accreditation  Plan  to  reflect  the  revised  Federation  Acceptability  Criteria 

•  Update  the  Federation  V&V  Plan  to  reflect  the  revised  Federation  Accreditation  Plan  and  Test  Plan, 
supporting  tools  and  test  strategies,  and  the  federation  results  sampling  strategy 

•  Contribute  to  updating  the  Federation  Test  Plan  from  the  revised  Federation  V&V  Plan 

•  Contribute  input  to  quality  assurance,  configuration  management,  data  collection,  management, 
and  analysis  plans  as  needed 

•  Recommend  tools  and  test  strategies  that  could  benefit  the  VV&A  efforts  and  products 

•  Determine  how  the  selected  tools  and  test  strategies  impact  VV&A  activities 

•  Check  that  the  tools  identified  for  execution  monitoring  and  control  can  support  the  planned  federation 
results  sampling  strategy 

•  Support  translating  Federation  Requirements  into  the  federation  execution  and  management  plans 

7. 3. 4. 3  Information  Produced 

•  Input  to  the  Federation  Development  and  Execution  Plan 

•  Input  to  the  Federation  Test  Plan  revisions 

•  Input  to  the  quality  assurance,  configuration  management,  data  collection,  management,  and  analysis 
plans  as  needed 

•  Input  to  the  federation  design,  development,  and  execution  tool  selection 

•  Revised  Federation  Accreditation  Plan  and  V&V  Plan 

7.4  Phase  4  -  Verify  Federation  Development  Products 

The  purpose  of  this  phase  in  the  overlay  is  to  support  the  federation’s  development.  This  begins  by  supporting 
the  development  of  and  contributing  to  verifying  the  FOM,  Federation  Agreements,  and  Federation 
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Infrastructure.  The  federation  VV&A  Team  also  supports  verifying  and  validating  the  federate 
implementations  when  needed  and  collects  the  information  produced  by  these  efforts.  Finally,  this  phase 
includes  the  activities  to  verify  and  validate  the  data  sets  needed  for  the  federation’s  execution. 

Figure  A 1-7  illustrates  the  key  activities  in  this  phase  of  the  VV&A  overlay.  The  subsections  that  follow 
describe  each  of  these  activities. 


Figure  A1-7:  Verify  Federation  Development  Products  (Phase  4)  Activity  Diagram. 


7.4.1  Activity  4.1  -  Support  Developing  Federation  Object  Model 

This  activity  supports  the  development  of  the  FOM.  The  FOM  defines  the  characteristics  of  those  data 
exchanges  needed  to  occur  between  participating  Federates  to  meet  the  Federation  Objectives.  The  goal  of 
FOM  development  is  to  unambiguously  assign  the  responsibilities  for  representing  the  entities,  actions, 
and  interactions  of  the  Federation  Conceptual  Model  to  the  assembled  group  of  federates. 
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No  single  “best”  way  to  develop  a  FOM  exists.  Several  different  approaches  have  been  used  and  all  have  their 
advantages  and  disadvantages.  The  choice  of  which  FOM  development  approach  to  use  depends  upon  the 
circumstances  of  the  individual  federation,  and  the  preferences  of  the  Federation  Development  Team. 
The  VV&A  Team  supports  the  Federation  Development  Team  by  analyzing  the  chosen  approach  to  determine 
the  impact  of  that  approach  on  the  planned  VV&A  processes  and  the  acceptability  of  the  federation  for  the 
intended  use. 

The  VV&A  Team  participates  as  part  of  the  Federation  Development  Team  in  the  FOM  development  effort 
by  assisting  in  the  review  of  existing  object  models  or  data  dictionaries  to  identify  reusable  object  models, 
or  object  model  elements,  and  by  assisting  in  the  selection  of  a  tool  or  tools  to  capture  and  document  the 
FOM.  They  also  ensure  that  the  strategy  used  to  allocate  the  responsibility  to  provide  parts  of  the  federation 
representations  to  individual  federates  does  not  adversely  impact  the  federation’s  validity.  Finally,  the  VV&A 
Team  should  document  the  assumptions  and  limitations  made  during  the  FOM  development  process. 
To  accomplish  this  task  the  VV&A  Team  will  require  access  to  the  products  generated  by  the  FOM 
development  effort. 

7. 4. 1.1  Information  Required 

•  Verified  and  validated  Federation  Conceptual  Model 

•  Verified  Federation  Design 

•  Existing  Federate  SOMs 

•  Federation  Development  and  Execution  Plan 

•  Existing  FOM  if  one  exists  or  is  being  reused  as  a  starting  point 

•  Supporting  resources  (e.g.,  object  model  development  tools,  object  model  libraries,  dictionaries) 

•  FOM  development  notes/agreements  (e.g.,  conceptual  model/scenario  to  FOM  mapping,  federate  role/ 
responsibility  mapping,  SOM-to-FOM  mapping) 

7. 4. 1.2  Functions  (Tasks) 

•  Support  choosing  a  FOM  development  approach  and  determine  the  implications  of  that  choice  upon 
subsequent  VV&A 

•  Support  identifying  appropriate  object  models  or  object  model  subsets  for  reuse 

•  Support  reviewing  applicable  data  dictionaries  to  identify  relevant  object  model  elements 

•  Assist  in  selecting  an  appropriate  tool  for  capturing  and  documenting  the  FOM 

•  Participate  in  the  FOM  design  to  ensure  that  the  representation  allocation  strategy  does  not  adversely 
impact  the  federation’s  validity 

•  Contribute  to  documenting  the  limitations  and  assumptions  of  the  FOM 

7. 4. 1.3  Information  Produced 

•  Contributions  to  the  FOM  development  effort 

•  Contributions  to  documentation  of  the  FOM  limitations  and  assumptions 
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7.4.2  Activity  4.2  -  Contribute  to  Verifying  Federation  Object  Model 

This  activity  verifies  the  FOM  against  the  verified  Federation  Design,  the  validated  Federation  Conceptual 
Model,  and  the  abilities  of  the  individual  federates  to  share  information,  and,  if  required,  indicates  the  need  for 
corrective  actions.  Any  information  exchanges  required  by  the  Federation  Design  or  needed  to  represent  the 
phenomena  defined  in  the  Federation  Conceptual  Model  that  are  not  reflected  in  the  FOM  suggest  missing 
objects  or  interactions.  An  incomplete  FOM  may  limit  the  federation’s  ability  to  achieve  the  federation 
objectives  documented  in  the  Federation  Objectives  Statement.  FOM  verification  also  adds  an  important  link 
in  the  traceability  chain. 

As  with  the  other  verification  activities,  the  VV&A  Team  should  approach  verifying  the  FOM  as  an  integral 
part  of  the  Federation  Development  Team  and  share  this  responsibility  as  required  by  the  specifics  of  the 
organizations  involved  and  the  delegations  of  responsibility.  Regardless  of  who  performs  the  FOM 
verification  tasks,  the  tasks  in  this  activity  assume  that  the  VV&A  Team  has  complete  access  to  the  FOM 
verification  results.  This  activity  assumes  that  verifying  the  FOM  can  be  automated  to  some  degree. 

7. 4. 2.1  Information  Required 

•  FOM 

•  Verified  and  validated  Federation  Conceptual  Model 

•  Verified  Federation  Design 

•  Verified  Federation  Scenarios 

•  Federation  V&V  Plan 

7. 4. 2. 2  Functions  (Tasks) 

•  Verify  the  internal  consistency  and  completeness  of  the  FOM 

•  Verify  the  consistency,  completeness,  and  correctness  of  the  FOM  against  the  Federation  Design, 
Federation  Conceptual  Model,  and  Federation  Scenarios 

•  Contribute  to  documenting  the  FOM  verification  results 

•  Contribute  to  documenting  the  traceability  between  the  FOM,  the  Federation  Conceptual  Model, 
and  Federation  Design 

7. 4. 2. 3  Information  Produced 

•  FOM  verification  results 

•  Information  describing  the  traceability  between  the  FOM  and  the  Federation  Conceptual  Model  and 
Federation  Design 

7.4.3  Activity  4.3  -  Support  Establishing  Federation  Agreements 

This  activity  supports  the  Federation  Development  Team  in  establishing  the  Federation  Agreements. 
Federation  Agreements  are  ad  hoc  agreements  between  Federation  Development  Team  members  that  are 
necessary  to  achieve  a  fully  consistent,  interoperable,  distributed  federation.  Federation  Agreements  are  made 
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to  resolve  federation  operational  design  and  execution  management  issues.  Potential  topics  for  Federation 
Agreements  include  federation  operational  procedures  necessitated  by  federation  selective  use  of  interface 
standard  services,  federate  use  of  common  databases  and  algorithms,  or  federation  and  federate  strategies  for 
data  collection.  The  Federation  Development  Team  is  responsible  for  explicitly  considering  what  additional 
agreements  are  required  and  how  those  agreements  should  be  documented.  As  part  of  this  activity,  the  VV&A 
Team  can  make  recommendations  for  agreements  that  impact  the  interoperability  between  the  federates’ 
representations. 

The  VV&A  Team  participates  as  part  of  the  Federation  Development  Team  to  assess  how  the  proposed 
Federation  Agreements  could  affect  the  federation’s  validity.  In  this  role,  they  should  carefully  examine  each 
agreement  to  determine  its  impact  upon  the  federation’s  representational  capabilities  and  the  federation’s 
ability  to  meet  the  Federation  Acceptability  Criteria.  The  VV&A  Team  should  ensure  that  the  agreements  to 
support  interoperability  between  the  federates’  representations  are  negotiated  and  documented.  To  accomplish 
this,  the  VV&A  Team  will  require  access  to  all  of  the  Federation  Agreements  created  during  the  federation 
development. 

In  addition,  the  VV&A  Team  may  need  to  establish  agreements  of  their  own  to  govern  planned  VV&A 
processes. 

7. 4. 3.1  Information  Required 

•  Verified  FOM 

•  Verified  Federation  Scenarios 

•  Verified  and  validated  Federation  Conceptual  Model 

•  Verified  Federation  Design 

•  Federation  Development  and  Execution  Plan 

•  Federation  Acceptability  Criteria 

•  Federation  Development  Team  informal  documentation  (e.g.,  notes  and  agreements) 

7. 4. 3. 2  Functions  (Tasks) 

•  Support  the  following  FEDEP  tasks: 

•  Deciding  the  behavior  of  all  federation  objects  and  how  they  should  interact  during  execution 
(to  maintain  validity) 

•  Identifying  the  necessary  software  modifications  to  selected  federates  to  assure  federation  validity 

•  Deciding  which  databases  and  algorithms  should  be  common  or  consistent  in  order  to  maintain 
sufficient  interoperability 

•  Identifying  authoritative  data  sources  for  federate  and  federation  databases  and  checking  their 
pedigrees 

•  Deciding  how  time  should  be  managed  in  the  federation  in  order  to  maintain  validity 

•  Establishing  synchronization  points  for  the  federation  and  the  procedures  for  federation  initiation  to 
assure  federation  validity  during  operation 
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*  Deciding  the  strategy  for  how  the  federation  shall  be  saved  and  restored  in  order  to  maintain 
sufficiently  valid  continuity 

•  Deciding  how  data  is  to  be  distributed  to  and  collected  from  across  the  federation 

*  Transforming  the  functional  scenario  descriptions  to  executable  scenarios 

•  Reviewing  security  agreements  and  procedures  to  ensure  that  these  procedures  do  not  adversely 
impact  validity  or,  at  least,  to  make  the  development  team  aware  of  impacts  upon  validity 

•  Ensure  that  agreements  to  support  interoperability  between  the  federates’  representations  are  negotiated 
and  documented 

7.4.33  Information  Produced 

•  Input  to  the  Federation  Agreements 

•  Input  to  the  Federation  Scenario  instances 

7.4.4  Activity  4.4  -  Contribute  to  Verifying  Federation  Agreements 

This  activity  verifies  the  consistency,  completeness,  and  correctness  of  the  Federation  Agreements  both 
internally  and  against  the  verified  FOM,  the  verified  Federation  Design,  the  validated  Federation  Conceptual 
Model,  and  the  verified  Federation  Scenarios.  This  verification  process  demonstrates  that  all  interoperability 
issues  critical  for  information  sharing  and  compatibility  of  the  selected  federates  not  covered  in  the  FOM  are 
addressed  and  indicates  the  need  for  corrective  actions  that  may  be  required. 

Special  care  should  be  taken  to  establish  agreements  to  ensure  the  consistency  of  internal  federate 
representations  (e.g.,  coordinate  systems,  line-of-sight  calculations)  within  the  federation,  but  that  are  not 
explicitly  represented  in  the  FOM,  the  Federation  Design  or  the  verified  Federation  Requirements.  The  absence 
of  important  Federation  Agreements  can  significantly  distort  combined  federation  representations. 
This  activity  also  documents  the  traceability  between  the  Federation  Agreements  and  the  Federation  Design, 
the  Federation  Requirements,  the  Federation  Conceptual  Model,  and  the  Federation  Scenarios. 

As  with  the  other  verification  activities,  the  VV&A  Team  should  approach  verifying  the  Federation 
Agreements  as  an  integral  part  of  the  Federation  Development  Team  and  share  this  responsibility  as  required 
by  the  specifics  of  the  organizations  involved  and  the  delegations  of  responsibility.  Regardless  of  who 
performs  Federation  Agreement  verification,  the  tasks  in  this  activity  assume  that  the  VV&A  Team  has 
complete  access  to  the  Federation  Agreement  verification  results.  In  addition,  this  activity  assumes  that  the 
Federation  Agreements  are  clearly  and  unmistakably  documented. 

7. 4. 4. 1  Information  Required 

•  Federation  Agreements 

•  Verified  Federation  Scenarios 

•  Verified  and  validated  Federation  Conceptual  Model 

•  Verified  Federation  Design 

•  Verified  FOM 

•  Federation  V&V  Plan 
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7. 4. 4. 2  Functions  (Tasks) 

•  Verify  the  internal  consistency  and  completeness  of  the  Federation  Agreements 

•  Verify  Federation  Agreement  consistency  with  the  FOM,  Federation  Design,  Federation  Conceptual 
Model,  and  Federation  Scenarios 

•  Contribute  to  documenting  the  Federation  Agreements  verification  results 

•  Contribute  to  documenting  the  traceability  between  the  Federation  Agreements  and  the  Federation 
Conceptual  Model  and  Federation  Design 

7. 4. 4. 3  Information  Produced 

•  Federation  Agreements  verification  results 

•  Information  describing  the  traceability  between  the  Federation  Agreements  and  the  Federation  Conceptual 
Model  and  Federation  Design 

7.4.5  Activity  4.5  -  Support  Implementing  Federate  Designs 

This  activity  supports  the  V&V  activities  associated  with  implementing  the  designs  of  the  individual  federates 
and  modifying  existing  federates  so  they  can  comply  with  the  Federation  Conceptual  Model,  the  FOM, 
and  the  Federation  Agreements.  The  federation  VV&A  Team  cooperates  with  the  V&V  teams  supporting  the 
individual  federate  design,  development,  and  modification  efforts  to  ensure  that  new  or  modified  federates  are 
verified  and  validated  as  needed  to  support  the  Federation  Objectives.  The  federation  VV&A  Team  must  also 
ensure  that  traceability  links  between  the  individual  federate  designs  and  the  Federation  Design  and 
Federation  Requirements  exist. 

The  federation  VV&A  Team  recommends  the  V&V  information  requirements  to  the  federate  V&V  teams. 
The  federation  VV&A  Team  then  collects  and  integrates  the  results  that  these  V&V  activities  produce. 
Information  and  associated  documentation  not  provided  by  the  federate  V&V  teams  adds  to  the  uncertainty 
associated  with  the  federation’s  validation. 

This  activity  assumes  that  a  close  cooperative  relationship  exists  between  the  federation  VV&A  Team  and  the 
members  of  the  V&V  teams  associated  with  the  individual  federates.  Under  some  circumstances, 
the  federation  VV&A  Team  may  need  to  take  a  more  active  role  in  federate  verification  and  validation,  but  the 
available  resources  may  limit  the  extent  of  this  level  of  participation.  This  may  range  from  simply  monitoring 
to  actually  performing  detailed  testing  on  the  individual  federate. 

7. 4. 5.1  Information  Required 

•  Verified  Federation  Design 

•  Verified  Federation  Agreements 

•  Verified  and  validated  Federation  Conceptual  Model 

•  Verified  Federation  Scenarios 

•  Verified  FOM 
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•  Verified  Federation  Requirements 

•  Federation  Acceptability  Criteria 

•  Federate  V&V  information 

7. 4. 5. 2  Functions  (Tasks) 

•  Collect  and  integrate  the  historical  and  newly  produced  V&V  information  describing  each  federate’s 
validated  representational  capabilities 

•  Recommend  V&V  information  requirements  for  new  or  modified  federates 

•  Contribute  to  verifying  the  mapping  of  new  or  modified  federate  capabilities  to  the  Federation  Design,  as 
needed 

•  Contribute  to  the  other  verification  activities  for  each  new  or  modified  federate,  as  needed 

•  Contribute  to  validating  the  federates  against  the  Federation  Acceptability  Criteria  and  the  Federation 
Referent,  as  needed 

•  Contribute  to  documenting  the  traceability  between  each  federate  and  the  Federation  Design,  Federation 
Scenarios,  Federation  Conceptual  Model,  Federation  Agreements,  FOM,  Federation  Requirements,  and 
Federation  Acceptability  Criteria 

7. 4. 5. 3  Information  Produced 

•  Integrated  historical  and  newly  produced  V&V  information  for  each  federate 

•  V&V  information  requirements  for  new  or  modified  federates 

•  Input  supporting  federate  verification  activities,  as  needed 

•  Input  supporting  federate  validation  activities,  as  needed 

•  Information  describing  the  federate  traceability,  as  needed 


7.4.6  Activity  4.6  -  Support  Implementing  Federation  Infrastructure 

In  this  activity,  the  VV&A  Team  supports  the  Federation  Development  Team  in  implementing,  configuring, 
and  initializing  the  infrastructure  necessary  to  realize  the  federation  and  ensure  that  that  infrastructure  can 
support  the  execution  and  intercommunication  of  all  of  the  federates.  This  involves  implementing  the  network 
design  (e.g.,  wide-area  networks  and  local-area  networks),  initializing  and  configuring  the  network  elements 
(e.g.,  routers  and  bridges),  and  installing  and  configuring  the  supporting  software  on  all  computer  systems. 
All  of  these  tasks  are  necessary  to  produce  a  successfully  operating  federation. 

A  federation’s  infrastructure  can  affect  the  validity  of  its  representations  in  numerous  ways.  Therefore, 
the  VV&A  Team  should  participate  as  part  of  the  Federation  Development  Team  in  implementing  those 
aspects  of  the  federation  infrastructure  that  could  affect  the  federation’s  validity.  In  this  role,  they  should 
check  that  the  federation’s  infrastructure  functions  sufficiently  well  to  ensure  the  federation’s  validity. 
Further,  the  infrastructure  (e.g.,  runtime  infrastructure)  initialization  data  may  also  be  able  to  influence 
federation’s  representational  capabilities.  As  a  result,  the  VV&A  Team  should  assure  that  this  data  does  not 
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adversely  affect  federation’s  validity.  The  FEDEP  states,  extensive  modification  to  the  RTI  initialization 
data  are  generally  unnecessary,  and  should  only  be  undertaken  with  sufficient  knowledge  of  their  potential 
impacts  on  the  federation  as  a  whole,  minor  modifications  can  improve  federation  performance  in  some 
circumstances.”  Thus,  the  VV&A  Team  should  ensure  that  any  modifications  to  the  infrastructure 
initialization  data  do  not  adversely  affect  the  validity  of  the  federation  as  a  whole. 

This  activity  assumes  that  the  VV&A  Team  can  readily  access  all  of  the  information  describing  the 
characteristics  of  the  federation’s  infrastructure. 

7. 4. 6. 1  Information  Required 

•  Federation  infrastructure  documentation  (implementation  and  configuration  management  plan; 
initialization  plan,  host/platform  information  tables,  network  information  tables,  etc.) 

•  Verified  Federation  Agreements 

•  Verified  Federation  Design 

•  Verified  FOM 

•  Federation  Development  and  Execution  Plan 

•  Federation  Acceptability  Criteria 

7. 4. 6. 2  Functions  (Tasks) 

•  Assist  in  determining  that  the  planned  infrastructure  design,  configuration,  and  initialization  will  not 
adversely  affect  the  ability  of  the  federation  to  meet  the  representational  requirements  defined  by  the 
Federation  Acceptability  Criteria 

7. 4. 6. 3  Information  Produced 

•  Input  to  the  implemented  federation  infrastructure  design 

7.4.7  Activity  4.7  -  Contribute  to  Verifying  Federation  Infrastructure 

This  activity  verifies  the  consistency,  completeness,  and  correctness  of  the  federation  infrastructure  both 
internally  and  against  the  verified  Federation  Agreements  and  verified  Federation  Design. 

This  verification  activity  demonstrates  that  the  infrastructure  supports  the  information  sharing  and  exchange 
among  the  federates  needed  to  meet  the  Federation  Requirements  and  suggests  the  need  for  any  corrective 
actions.  If  the  federation  infrastructure  cannot  provide  the  needed  quality  of  service  to  support  the  federate 
interactions  then  the  federation’s  validity  may  be  compromised  so  that  it  cannot  achieve  the  Federation 
Objectives. 

As  with  the  other  verification  activities,  the  VV&A  Team  should  approach  verifying  the  federation 
infrastructure  as  an  integral  part  of  the  Federation  Development  Team  and  share  this  responsibility  as  required 
by  the  specifics  of  the  organizations  involved  and  the  delegations  of  responsibility.  Regardless  of  who  verifies 
the  federation  infrastructure,  the  tasks  in  this  activity  assume  that  the  VV&A  Team  has  complete  access  to  the 
infrastructure  verification  results. 
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7. 4. 7. 1  Information  Required 

•  Federation  infrastructure  documentation  (implementation  and  configuration  management  plan;  initialization 
plan,  host/platform  information  tables,  network  information  tables,  etc.) 

•  Federation  infrastructure  design 

•  Verified  Federation  Agreements 

•  Verified  Federation  Design 

•  Verified  FOM 

•  Federation  Development  and  Execution  Plan 

•  Federation  Acceptability  Criteria 

7. 4. 7. 2  Functions  (T asks) 

•  Participate  in  verifying  that  the  federation  infrastructure  design  and  implementation  are  consistent  with 
the  Federation  Design  and  development  plans  and  will  meet  the  documented  infrastructure  requirements 

•  Contribute  to  verifying  that  correct  configuration,  initialization,  and  operation  of  the  infrastructure  has 
been  achieved 

•  Contribute  to  verifying  that  all  federates  have  properly  adhered  to  the  infrastructure  requirements  and 
associated  configurations  and  initializations 

•  Contribute  to  verifying  that  the  implemented  infrastructure  supports  the  execution  and  intercommunication 
of  the  federates 

•  Contribute  to  documenting  the  federation  infrastructure  design  results 

•  Contribute  to  documenting  the  traceability  between  federation  infrastructure  design  and  the  Federation 
Design,  FOM,  and  the  Federation  Agreements 

•  Observe  infrastructure  testing  done  by  the  Federation  Development  Team 

•  Determine  if  the  infrastructure  design,  implementation,  configuration,  and  initialization  have  affected  the 
ability  of  the  federation  to  meet  the  Federation  Acceptability  Criteria 

7. 4. 7. 3  Information  Produced 

•  Federation  infrastructure  design  verification  results 

•  Information  describing  the  traceability  between  federation  infrastructure  design  and  the  FOM,  Federation 
Design  and  Federation  Agreements 

•  Observations  of  the  federation  infrastructure  testing 

•  Determination  of  how  the  infrastructure  design  and  implementation  affect  the  federation’s  representational 
characteristics 

7.4.8  Activity  4.8  -  Verify  and  Validate  Federation  Data  Sets 

A  federation  uses  a  combination  of  data  sets,  each  created  for  a  specific  purpose.  Some  data  sets  are  used  by 

individual  federates.  Two  or  more  federates  may  share  data  or  common  databases  while  other  data  sets  pertain 
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to  the  entire  federation  execution.  Finally,  data  may  have  been  created  to  fill  voids  in  available  databases  or  to 
support  federation  testing.  Regardless  of  whether  federates  and  data  sets  have  been  used  together  in  previous 
federations,  some  data  set  verification  and  validation  will  be  necessary.  The  amount  will  depend  upon  the 
Federation  Objectives,  the  acceptable  risk,  and  the  quality  of  the  history  available  on  each  of  the  federates, 
and  the  data  sets  that  they  need.  Any  new  data  generated  for  the  federation  should  be  validated. 

Data  verification  and  validation  are  necessary  at  the  federation  level  to  ensure  that  the  selected  data  sets  have 
been  obtained  from  appropriate  sources,  they  are  sufficiently  correct  to  support  the  intended  use,  and  the 
federates  can  meaningfully  use  them.  This  means  that  the  data  contained  by  those  data  sets  have  correct  units 
of  measure,  have  values  within  the  federate  input  limits,  are  internally  consistent,  and  are  measured  within  the 
coordinate  systems  employed  within  the  federates  or  are  converted,  as  necessary.  If  a  federation  data  model 
was  constructed  as  part  of  Federation  Design,  it  should  be  compared  to  the  data  set  characteristics  to  assure 
the  completeness  of  those  data  sets  and  their  consistency  with  the  Federation  Design. 

As  with  the  other  verification  activities,  the  VV&A  Team  should  approach  verifying  the  federation  data  sets 
as  an  integral  part  of  the  Federation  Development  Team  and  share  this  responsibility  as  required  by  the 
specifics  of  the  organizations  involved  and  the  delegations  of  responsibility.  Regardless  of  who  verifies 
the  federation  data  sets,  the  tasks  in  this  activity  assume  that  the  VV&A  Team  has  complete  access  to  the  data 
set  verification  results.  This  activity  also  assumes  that  data  set  validation  precedes  federation  validation. 
Testing  should  be  conducted  during  federation  construction  and  assembly  to  take  advantage  of  ongoing 
federate  compatibility  tests. 

7. 4. 8. 1  Information  Required 

•  Federation  Acceptability  Criteria 

•  Federation  Referent 

•  Federation  data  sets  and  their  metadata  descriptions 

•  Federation  data  set  histories 

•  Federation  data  set  use  restrictions 

•  Federation  Design 

•  Federate  algorithm  descriptions 

•  Federation  V&V  Plan 

7. 4. 8. 2  Functions  (Tasks) 

•  Verify  data  set  sources  to  ensure  the  data  and  database  sources  are  the  most  appropriate  available 

•  Verify  that  federates  are  capable  of  exchanging  data  and  sharing  the  databases  as  specified  in  the 
Federation  Design 

•  Verify  that  the  federate  algorithms  consistently  support  transformation  and  aggregation  of  the  common 
data  sets 

•  Compare  key  source  data  and  metadata  to  federate  input  values  and  specifications  to  verify  values 
assigned 
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•  Identify  differences  in  format  or  content  of  the  key  source  data  and  metadata 

•  Assess  the  impact  of  any  discovered  discrepancies  upon  federation  execution  and  output 

•  Contribute  to  documenting  the  traceability  of  inputs  through  applied  transformations  to  ensure  that  the 
data  have  appropriate  values  and  are  handled  consistently 

•  Compare  the  data  sets  to  the  Federation  Acceptability  Criteria  to  ensure  the  data  selected  are  the  data 
desired 

•  Ensure  that  the  data  needed  are  available  and  data  voids  and  deficiencies  are  identified 

•  Validate  data  sets  against  applicable  Federation  Acceptability  Criteria 

•  Conduct  sensitivity  analyses  to  assess  the  impact  of  data  variations  on  the  output  results 

•  Assess  the  impacts  and  risks  associated  with  inconsistent,  incomplete,  or  incorrect  data  sets  and  make 
recommendations  for  reducing  those  risks 

•  Document  the  results  of  data  set  verification  and  validation 

7. 4. 8. 3  Information  Produced 

•  Federation  data  set  validation  results 

•  Federation  data  set  verification  results 

•  Information  describing  the  traceability  between  the  federation  data  sets,  their  sources,  and  their 
transformations  within  the  federates 

•  Data  variation  sensitivity  analysis  results 

•  Data  set  incompleteness  and  incorrectness  risk  assessment 

7.5  Phase  5  -  Validate  and  Accept  Federation 

The  primary  purpose  of  Phase  5  of  the  VV&A  overlay  is  to  validate  and  accept  the  federation  for  its  intended 
use.  This  phase  begins  by  supporting  the  execution  planning  in  order  to  assure  that  the  testing  process  will 
provide  sufficient  results  to  enable  validating  the  integrated  federation.  The  VV&A  team  also  supports  the 
federation  integration  activities  of  the  FEDEP  as  needed  and  contributes  to  verifying  the  integrated  federation. 
Additionally,  they  support  federation  testing  and  use  the  results  from  that  testing  to  validate  the  integrated 
federation.  The  results  from  all  of  these  and  all  previous  V&V  activities  supply  the  evidence  upon  which  to 
base  the  federation  acceptance/accreditation  recommendations. 

Figure  A 1-8  illustrates  the  key  activities  in  this  phase  of  the  VV&A  overlay.  The  subsections  that  follow 
describe  each  of  these  activities. 
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Figure  A1-8:  Validate  and  Accept  Federation  (Phase  5)  Activity  Diagram. 


7.5.1  Activity  5.1  -  Support  Planning  Federation  Execution 

In  this  activity,  the  VV&A  team  supports  the  Federation  Development  Team  in  planning  of  the  federation’s 
execution.  This  final  planning  activity  involves  several  tasks  that  could  impact  the  federation’s 
representations.  For  example,  the  procedures  for  interrupting  and  restarting  the  federation  execution  could 
affect  the  federation’s  representation  of  causality  and  that  could  change  the  validity  of  the  execution’s  results. 
The  VV&A  Team  should  work  within  the  Federation  Development  Team  to  minimize  the  impact  of  any 
execution  planning  decisions  upon  the  execution’s  validity. 

This  planning  activity  refines  and  augments  the  Federation  Development  and  Execution  Plan.  The  VV&A 
Team  should  identify  the  impacts  that  these  changes  could  have  upon  the  federation’s  validity  and  suggest 
modifications  that  could  minimize  these  impacts  where  possible.  The  VV&A  Team  may  also  need  to  refine 
the  Federation  Accreditation  Plan  and  V&V  Plan  to  accommodate  these  changes  and  to  incorporate  any 
additional  information  gathered.  Further,  the  VV&A  Team  should  participate  as  part  of  the  Federation 
Development  Team  in  identifying  risks  and  the  actions  needed  to  ameliorate  them. 

The  VV&A  Team  should  provide  any  additional  inputs  to  the  Federation  Development  and  Execution  Plan 
and  its  subordinate  test  and  data  collection  plans  that  ensure  that  the  data  they  need  for  results  validation  will 
be  collected  under  the  execution  conditions  described  by  the  results  sampling  strategy  and  design  of 
experiments.  The  VV&A  Team  should  exploit  all  of  the  executions  as  much  as  possible  to  optimize  the  results 
collected. 

This  activity  assumes  that  the  VV&A  Team  has  access  to  all  of  the  information  supporting  the  development 
and  modification  of  the  Federation  Development  and  Execution  Plan. 
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7. 5. 1.1  Information  Required 

•  Verified  FOM 

•  FOM  Document  Data/Federation  Execution  Data 

•  Federation  Scenario  instances 

•  Verified  Federation  Agreements 

•  Federation  Development  and  Execution  Plan 

•  Federation  Accreditation  Plan  and  V&V  Plan 

7. 5. 1.2  Functions  (Tasks) 

•  Support  identifying  risks  and  the  actions  to  address  them 

•  Support  documenting  all  of  the  information  relevant  to  the  federation  execution 

•  Support  developing  detailed  execution  plans  especially  those  relevant  to  collecting  data  to  support 
federation  results  validation  (e.g.,  testing  and  data  collection)  and  those  that  might  affect  federation 
validity  (e.g.,  security) 

•  Assess  the  effects  that  the  special  procedures  for  starting,  stopping,  and  controlling  each  execution  have 
on  the  validity  of  the  federation  execution  and  document  these  effects 

•  Refine  the  Federation  Accreditation  Plan  and  V&V  Plan  and  provide  these  modifications  to  the  Federation 
Development  and  Execution  Plan 

•  Evaluate  the  impacts  that  other  modifications  to  the  Federation  Development  and  Execution  Plan  may 
have  upon  federation  validity  (e.g.,  security  plan)  and  provide  this  as  feedback  to  the  Federation 
Development  Team 

7. 5. 1.3  Information  Produced 

•  Input  to  federation  risk  management  activities 

•  Input  to  federation  execution  documentation 

•  Input  to  the  Federation  Execution  Environment  Description 

•  Revised  Federation  Accreditation  Plan  and  V&V  Plan 

•  Modifications  to  the  Federation  Development  and  Execution  Plan  due  to  Federation  Accreditation  Plan 
and  V&V  Plan  revisions 

•  Evaluations  of  the  impact  of  changes  to  the  Federation  Development  and  Execution  Plan  that  could 
impact  federation  validity 

7.5.2  Activity  5.2  -  Support  Integrating  Federation 

In  this  activity,  the  VV&A  Team  supports  the  Federation  Development  Team  in  integrating  the  federation  to 

assure  that  the  integration  process  results  in  valid  federation  representations.  This  may  involve  monitoring  the 
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federation  integration  process  and  observing  that  the  Federation  Development  and  Execution  Plan  is  followed, 
particularly  those  parts  that  could  impact  the  federation’s  validity. 

The  Federation  Development  Team  may  encounter  and  resolve  problems  during  integration.  The  VV&A 
Team  should  determine  the  impacts  that  these  problem  resolutions  have  upon  the  federation’s  representations 
and  their  ability  to  meet  the  Federation  Acceptability  Criteria.  This  will  help  to  avoid  workarounds  that 
adversely  affect  federation  validity  and  will  improve  the  likelihood  of  producing  a  valid  federation  for  the 
users. 

This  activity  assumes  that  the  VV&A  Team  has  complete  access  to  all  aspects  of  the  federation  integration 
process. 

7. 5. 2.1  Information  Required 

•  Federation  Development  and  Execution  Plan 

•  Verified  federation  development  products 

•  Federation  Execution  Environment  Description 

•  Verified  and  validated  Federation  Conceptual  Model 

•  Verified  Federation  Design 

•  Verified  Federation  Agreements 

•  Verified  FOM 

•  Runtime  infrastructure  initialization  data 

•  Verified  and  validated  federates  (existing  selected,  modified,  newly  developed  federates) 

•  Implemented  and  verified  federation  infrastructure  design 

•  Verified  and  validated  supporting  data  sets 

7. 5. 2. 2  Functions  (Tasks) 

•  Determine  the  impact  of  the  methods  for  managing  known  hardware,  interface,  and  software  problems 
and  the  “workarounds”  to  overcome  them  upon  federation  validity 

•  Determine  the  impact  of  deviations  from  the  Federation  Development  and  Execution  Plan,  particularly 
those  aspects  that  could  affect  federation  validity 

•  Monitor  the  conduct  of  the  federation  integration  process 

7. 5. 2. 3  Information  Produced 

•  Assessment  of  the  impact  of  hardware,  interface,  and  software  problems  and  their  workarounds  upon 
federation  validity 

•  Assessment  of  the  impact  of  deviations  from  the  Federation  Development  and  Execution  Plan  upon 
federation  validity 
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7.5.3  Activity  5.3  -  Contribute  to  Verifying  Integrated  Federation 

This  activity  verifies  the  consistency,  completeness,  and  correctness  of  the  integrated  federation  both 
internally  and  against  the  verified  Federation  Agreements,  the  verified  Federation  Design,  and  the  validated 
Federation  Conceptual  Model. 

This  verification  activity  demonstrates  that  both  the  federates  and  the  integrated  federation  function  as 
designed  and  as  needed  to  meet  the  Federation  Requirements.  For  example,  this  activity  should  verify  that  all 
attribute  updates  and  interactions  are  sent  and  received  from  source  to  destination  as  designed.  In  addition, 
the  assertions,  pre-  and  post-conditions  as  well  as  behavior  constraints  described  in  the  Federation  Design 
should  be  verified.  This  verification  activity  should  also  assess  whether  the  federate  hosts  can  communicate 
with  each  other  and  whether  the  network  capacity  and  resulting  communications  latency  meets  the  design 
specification.  Failure  to  conform  to  the  federate  or  federation  design  specifications  suggests  that  the  federation 
cannot  meet  the  Federation  Acceptability  Criteria.  This  situation  indicates  the  need  for  any  corrective  actions. 

As  with  the  other  verification  activities,  the  VV&A  Team  should  approach  verifying  the  integrated  federation 
as  part  of  the  Federation  Development  Team  and  share  this  responsibility  as  required  by  the  specifics  of  the 
organizations  involved  and  the  delegations  of  responsibility. 

Regardless  of  who  verifies  the  federation  infrastructure,  the  tasks  in  this  activity  assume  that  the  VV&A  Team 
has  complete  access  to  the  infrastructure  verification  results.  This  activity  also  assumes  that  tools  for 
monitoring  federation  traffic  are  available. 

7. 5. 3.1  Information  Required 

•  Verified  Federation  Design 

•  Verified  FOM 

•  Verified  and  validated  Federation  Conceptual  Model 

•  Verified  Federation  Agreements 

•  Integrated  federation 

•  Verified  Federation  Test  Criteria 

•  Federation  V&V  Plan 

7. 5. 3. 2  Functions  (Tasks) 

•  Contribute  to  verifying  that  the  federation  behaves  according  to  the  Federation  Design 

•  Contribute  to  verifying  that  the  integrated  federation  as  a  whole  behaves  according  to  the  FOM, 
Federation  Conceptual  Model,  and  Federation  Agreements 

•  Contribute  to  verifying  that  adequate  communication  exists  to  support  valid  federation  operation 

•  Verify  that  federation  tests  adhered  to  the  Federation  Test  Plan 

•  Contribute  to  documenting  the  integrated  federation  verification  results 

•  Contribute  to  documenting  the  traceability  between  the  integrated  federation  and  the  Federation  Design, 
FOM,  and  the  Federation  Agreements 
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7. 5. 3. 3  Information  Produced 

•  Integrated  federation  verification  results 

•  Input  on  whether  the  federation  tests  adhered  to  the  Federation  Test  Plan 

•  Information  describing  the  traceability  between  the  integrated  federation  and  the  FOM,  Federation 
Design,  and  Federation  Agreements 

7.5.4  Activity  5.4  -  Support  Testing  Federation 

In  this  activity,  the  VV&A  Team  supports  the  Federation  Development  Team  in  testing  the  federation.  All  of 
the  phases  of  this  federation  testing  can  supply  data  that  is  critical  to  results  validation.  Thus,  the  VV&A 
Team  needs  to  participate  in  the  testing  to  understand  the  actual  test  conditions  in  order  to  correctly  inteipret 
the  test  data  for  federation  validation. 

The  VV&A  Team  should  ensure  that  testing  produces  the  data  as  defined  in  the  test  and  data  collection  plans 
and  that  the  test  data  is  stored  in  an  accessible  form.  They  should  also  determine  the  impacts  that  any 
corrective  actions  planned  by  the  Federation  Development  Team  have  upon  the  federation’s  validity. 
This  information  may  identify  the  need  to  change  these  corrective  action  or  modify  the  Federation  Scenarios, 
Federation  Objectives  or  Federation  Acceptability  Criteria.  The  VV&A  Team  should  modify  the  Federation 
Accreditation  Plan  and  V&V  Plan  to  reflect  these  changes. 

This  activity  assumes  that  the  VV&A  Team  has  access  to  all  of  the  data  produced  by  the  federation  testing 
tasks  and  that  the  testing  tasks  execute  the  test  and  data  collection  plans  so  as  to  produce  sufficient  data  for 
results  validation. 

7. 5. 4.1  Information  Required 

•  Verified  integrated  federation 

•  Federation  Development  and  Execution  Plan 

•  Verified  Federation  Agreements 

•  Federation  Execution  Environment  Description 

•  Verified  Federation  Test  Criteria 

•  HLA  Federate  Interface  Specification,  Framework  and  Rules,  and  Object  Model  Template  Specification 

7. 5. 4. 2  Functions  (Tasks) 

•  Support  federate-level  testing 

•  Support  federation-level  and  interoperability  testing 

•  Observe  federation  testing  activities  and  collect  data  from  these  activities  needed  to  support  federation 
results  validation 
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7. 5. 4. 3  Information  Produced 

•  Input  to  federate  and  federation  testing 

•  Federation  testing  results  to  support  federation  results  validation 

•  Data  and  observations  collected  during  federation  testing 

7.5.5  Activity  5.5  -  Validate  Integrated  Federation  Results 

Results  validation  is  the  pinnacle  of  the  federation  validation  effort.  Results  validation  assesses  by  checking 
the  validity  of  the  actual  federation  output  against  the  Federation  Acceptability  Criteria.  While  results 
validation  is  only  one  piece  of  federation  validation  evidence,  it  often  has  the  most  credibility  with  the  User/ 
Sponsor. 

Federation  results  validation  builds  upon  the  federation  testing  efforts  and  may  require  additional  federation 
executions  to  augment  the  federation  testing  results.  The  results  sampling  strategy  determines  the  need  for 
these  added  executions  as  well  as  the  required  content  from  the  tests.  This  activity  should  also  determine  the 
error  characteristics  and  uncertainties  associated  with  the  sampled  data  and  the  results  from  any  additional 
executions.  This  activity  ends  by  inferring  the  root  causes  of  any  unacceptable  results.  This  unacceptability 
may  come  from  either  federation  behavior  or  the  testing  instrumentation  and  conditions.  In  either  case, 
these  causes  are  fed  back  to  the  Federation  Development  Team  for  correction.  This  feedback  may  result  in 
additional  testing  which  can  then  supplement  the  validation  results. 

This  activity  assumes  that  those  responsible  for  testing  the  federation  actually  perform  all  executions  and 
collect  the  results  from  those  executions.  This  activity  also  assumes  that  the  VV&A  Team  also  evaluates  those 
results  to  develop  the  evidence  to  support  federation  acceptance.  Further,  this  activity  assumes  that  a  results 
sampling  strategy  has  been  formulated  that  can  provide  sufficient  results  to  achieve  the  desired  confidence  in 
the  validation  evidence,  that  the  test  plan  has  incorporated  that  strategy,  and  that  the  plan  has  been  faithfully 
executed.  It  also  assumes  that  the  federation  results  provided  as  input  to  results  validation  come  from  a 
correctly  executing  integrated  federation  and  can  be  considered  as  representative  of  the  results  that  the  actual 
federation  User/Sponsor  will  see. 

7. 5. 5. 1  Information  Required 

•  Federation  Acceptability  Criteria 

•  Federation  Referent 

•  Federation  testing  results 

•  Functioning  integrated  federation  together  with  sufficient  data  to  perform  the  validation  executions 

•  Federation  V&V  Plan 

7. 5. 5. 2  Functions  (Tasks) 

•  Identify  executions  needed  to  supplement  those  performed  in  integration  testing  in  order  to  execute  the 
federation  results  sampling  strategy 

•  Perform  any  supplemental  federation  executions  necessary 


RTO-TR-MSG-019 


A1  -55 


APPENDIX  1  -  VV&A  OVERLAY  TO  THE  FEDEP 


ORGANIZATION 


•  Collect  federation  execution  results  data  and  correlate  them  into  a  integrated  picture  of  federation 
representational  behavior 

•  Estimate  the  uncertainties  associated  with  the  correlated  federation  results  sample 

•  Evaluate  the  completeness  of  the  federation  results  against  the  Federation  Acceptability  Criteria  and 
Federation  Referent  and  identify  areas  of  incompleteness 

•  Estimate  the  error  characteristics  of  the  correlated  federation  results 

•  Evaluate  the  correctness  of  the  correlated  federation  results  against  the  Federation  Acceptability  Criteria 
and  Federation  Referent  and  identify  areas  of  incorrectness 

•  Estimate  the  uncertainties  associated  with  the  errors  in  the  federation  results 

•  Identify  any  observed  interoperability  anomalies  and  infer  their  probable  causes 

•  Document  the  results  of  the  federation  results  validation  tasks 

7. 5. 5. 3  Information  Produced 

•  Federation  results  validation  results 

•  Within  the  context  of  the  Federation  Referent,  identification  of  the  Federation  Acceptability  Criteria 
that  the  federation  results 

•  Meets 

•  Do  not  meet 

•  Characterization  of  the  representational  errors  in  the  federation  results  that  exceed  the  Federation 
Acceptability  Criteria  limits 

•  Estimates  of  uncertainties  associated  with  the  representational  errors  in  the  federation  results 

•  Observed  interoperability  anomalies  and  their  probable  causes 

7.5.6  Activity  5.6  -  Perform  Acceptance  Assessment 

This  activity  results  in  recommendations  on  the  conditions  under  which  the  federation  results  can  serve  the 
User/Sponsor  Needs.  These  recommendations  guide  the  User/Sponsor’s  acceptance  of  the  integrated 
federation  for  their  intended  uses. 

In  this  activity,  the  VV&A  Team  collects  and  aggregates  all  of  the  V&V  evidence  and  derives  from  this 
evidence  the  federation  acceptance/accreditation  recommendations.  The  aggregation  process  should  draw 
integrated  assessments  from  disparate  V&V  results.  For  example,  one  piece  of  evidence  may  come  from  a 
subject  matter  expert’s  opinion  while  another  may  come  from  a  statistical  comparison  test.  The  means  to  deal 
with  these  differences  depend  completely  upon  the  characteristics  of  the  evidence  and  the  specific  nature  of 
the  Federation  Acceptability  Criteria. 

The  VV&A  Team  should  determine  the  degree  to  which  the  federation’s  capabilities  meet  the  Federation 
Acceptability  Criteria.  This  determination  should  identify  the  use  conditions  under  which  the  federation  meets 
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the  Federation  Acceptability  Criteria.  It  should  also  identify  those  criteria  that  the  federation  cannot  meet 
under  any  conditions  of  use.  The  Federation  Development  Team  can  deal  with  unmet  acceptability  criteria  in 
many  ways  including  revising  the  Federation  Scenarios,  modifying  the  federates  or  their  databases, 
or  modifying  the  Federation  Objectives,  thus  changing  the  federation’s  intended  use.  Some  of  these  solutions 
may  require  revisiting  parts  of  the  federation  development  process  and  its  associated  VV&A  activities. 

The  FEDEP  suggests  that  the  federation  testing  activity  should  result  in  an  integrated,  tested,  validated,  and  if 
required,  accredited  federation.  This  activity  only  accounts  for  the  VV&A  Team’s  support  for  producing  an 
integrated  and  tested  federation.  Verification,  validation,  and  acceptance  of  the  integrated  federation  are  all 
described  under  separate  subsequent  activities. 

The  recommendations  from  the  acceptance  assessment  may  lead  to  an  accreditation  decision  that  may  be 
issued  at  any  time  following  this  acceptance  assessment. 

7. 5. 6. 1  Information  Required 

•  Documentation  of  the  V&V  activities  and  their  results 

•  Federation  Acceptability  Criteria 

•  Federation  V&V  Plan 

•  Federation  Accreditation  Plan 

•  Federation  Impact  Use  Assessment 

•  FEDEP  products,  necessary  to  interpret  the  conducted  V&V-activities,  e.g.,  Federation  Objectives, 
Federation  Test  Criteria,  Federation  Development  and  Execution  Plan 

•  Federation  test  results 

7. 5. 6. 2  Functions  (Tasks) 

•  Determine  that  the  V&V  activities  adhered  to  the  constraints  of  the  Federation  V&V  Plan 

•  Gather  and  aggregate  the  information  produced  by  the  V&V  activities  during  the  preceding  parts  of  the 
federation  development  and  execution  process 

•  Evaluate  and  merge  the  V&V  results 

•  Estimate  the  uncertainties  associated  with  the  merged  V&V  evidence 

•  Estimate  the  risks  associated  with  using  the  federation  to  support  the  User/Sponsor  Needs  from  the 
impacts  described  in  the  Federation  Impact  Use  Assessment  and  the  uncertainties  associated  with  the 
merged  V&V  evidence 

•  Determine  the  recommended  conditions  of  federation  use  in  the  context  of  the  User/Sponsor  Needs 

•  Document  the  recommended  conditions  of  federation  use  and  their  rationale 

7. 5. 6. 3  Information  Produced 

•  Federation  acceptance/accreditation  recommendations 
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7.6  Phase  6  -  Verify  and  Validate  Federation  Output 

Phase  6  of  the  VV&A  overlay  supports  federation  execution  and  monitors  and  documents  the  occurrence  of 
any  execution  problems  that  may  affect  federation  validity.  These  problems  may  be  identified  through 
verification  of  the  raw  execution  and  derived  output  or  validation  of  the  derived  output. 

Figure  Al-9  illustrates  the  key  activities  in  this  phase  of  the  VV&A  overlay.  The  subsections  that  follow 
describe  each  of  these  activities. 


7.6.1  Activity  6.1  -  Support  Executing  Federation 

During  this  activity,  the  VV&A  Team  should  monitor  that  the  various  aspects  of  federation  execution  conform 
to  the  Federation  Development  and  Execution  Plan  and  that  any  occurrences  of  execution  problems  that  effect 
federation  validity  are  identified  and  recorded. 

Federation  executions  should  be  monitored  to  determine  how  well  the  Federation  Acceptability  Criteria  are 
met  in  order  to  provide  immediate  feedback  on  the  execution  of  the  federation.  If  the  VV&A  Team  has  not 
completely  validated  the  federation  by  this  phase,  additional  data  should  be  collected  to  further  assess  the 
validity  of  the  federation.  If  new  scenario  instances  or  input  data  are  introduced  at  this  stage,  the  VV&A  Team 
should  assess  the  significance  of  those  changes  and  provide  feedback  on  the  validity  implications. 
These  additions  to  the  federation  may  require  returning  to  the  previous  FEDEP  and  overlay  steps. 

If  federation  execution  problems  occur,  the  VV&A  Team  should  assess  the  impacts  of  these  problems  and 
their  workarounds,  if  any,  upon  the  validity  of  the  federation’s  results.  They  may  need  to  suggest  additional 
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constraints  upon  federation  use  to  maintain  the  execution  results  validity.  This  activity  assumes  that  some 
personnel  from  the  VV&A  Team  continue  to  be  involved  in  the  federation  execution  and  use  in  the  later  steps 
of  the  FEDEP. 

7. 6. 1.1  Information  Required 

•  Validated  and  accepted  integrated  federation 

•  Federation  Development  and  Execution  Plan 

•  Verified  Federation  Scenarios 

•  Federation  scenario  instances 

•  Verified  Federation  Agreements 

•  Federation  Execution  Environment  Description 

•  Federation  input  data 

•  Supporting  databases 

•  Federation  Acceptability  Criteria 

•  Federation  acceptance/accreditation  recommendations 

7. 6. 1.2  Functions  (Tasks) 

•  Monitor  that  the  federation  execution  conforms  to  the  Federation  Development  and  Execution  Plan, 
Federation  Scenarios,  Federation  Execution  Environment  Description,  and  Federation  Agreements 

•  Identify  the  occurrence  of  execution  problems  that  impact  the  validity  of  the  federation  execution  results 

7. 6. 1.3  Information  Produced 

•  Identification  of  any  execution  excursions  outside  the  federation  acceptance/accreditation  recommendations 

•  Inputs  to  the  documented  execution  problems 

7.6.2  Activity  6.2  -  Contribute  to  Verifying  Raw  Execution  and  Derived  Output 

This  activity  contributes  to  verifying  the  raw  data  collected  during  the  federation  execution  and  the  output 
derived  from  this  data  in  accordance  with  the  V&V  Plan  and  the  Federation  Development  and  Execution  Plan. 
This  includes  an  assessment  of  any  data  transformation  or  pre-processing  algorithms  or  techniques  that  have 
been  used.  In  addition,  any  incomplete,  inconsistent,  incorrect  or  erroneous  output  should  be  identified  and 
corrective  action  taken  where  necessary. 

During  this  activity,  quality  assurance/quality  management  tasks  may  be  ongoing.  If  so,  the  VV&A  Team 
should  collect  their  information  and  use  it  to  support  this  activity. 

This  activity  assumes  that  some  personnel  from  the  VV&A  Team  continue  to  be  involved  in  the  federation 
execution  and  use  in  the  later  steps  of  the  FEDEP  and  that  any  verification  techniques  used  conform  to  the 
V&V  Plan. 
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7.6.2. 1  Information  Required 

•  Raw  execution  output 

•  Derived  federation  output 

•  Federation  Development  and  Execution  Plan 

7. 6. 2. 2  Functions  (Tasks) 

•  Verify  that  the  raw  execution  output  conforms  to  data  collection  plan  that  is  part  of  the  Federation 
Development  and  Execution  Plan 

•  Verify,  with  the  users,  that  the  data  formats  and  transformations  of  the  federation  output  are  appropriate 
and  applied  within  their  constraints 

•  Identify,  with  the  user,  missing  and  erroneous  output 

•  Contribute  to  documenting  the  raw  execution  output  and  derived  federation  output  verification  results 

7. 6.2. 3  Information  Produced 

•  Raw  execution  output  verification  results 

•  Derived  federation  output  verification  results 

•  Information  on  any  missing  or  erroneous  output 

7.6.3  Activity  6.3  -  Validate  Federation  Output 

This  activity  validates,  against  the  Federation  Acceptability  Criteria  and  the  Federation  Referent, 
the  completeness  and  correctness  of  the  derived  output  data  collected  during  the  federation  execution  prior  to 
its  formal  analysis.  Any  areas  of  incompleteness  or  incorrectness  should  be  documented  and  reported  to  the 
User/Sponsor  and  the  Federation  Development  Team  together  with  the  uncertainties  associated  with  the 
derived  output  errors.  This  activity  builds  upon  prior  federation  results  validation  and  provides  an  additional 
opportunity  to  assess  the  validity  of  the  federation’s  output. 

This  activity  assumes  that  some  personnel  from  the  VV&A  Team  continue  to  be  involved  in  the  federation 
execution  and  use  in  the  later  steps  of  the  FEDEP  and  that  the  validation  techniques  selected  for  this  activity 
conform  to  the  V&V  Plan. 

7.6.3. 1  Information  Required 

•  Federation  Development  and  Execution  Plan 

•  Verified  derived  federation  output 

•  Verified  Federation  Agreements 

•  Documentation  of  execution  problems 

•  Federation  Acceptability  Criteria 
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•  Federation  Referent 

•  Federation  acceptance/accreditation  recommendations 

7. 6. 3. 2  Functions  (Tasks) 

•  Evaluate  the  completeness  of  the  derived  federation  output  against  the  Federation  Acceptability  Criteria 
and  Federation  Referent  and  identify  areas  of  incompleteness 

•  Estimate  the  error  characteristics  of  the  derived  federation  output 

•  Evaluate  the  correctness  of  the  derived  federation  output  against  the  Federation  Acceptability  Criteria  and 
Federation  Referent  and  identify  areas  of  incorrectness 

•  Estimate  the  uncertainties  associated  with  the  errors  in  the  derived  federation  output 

•  Identify  any  observed  representational  anomalies  and  infer  their  probable  causes 

•  Document  the  results  of  validating  the  derived  federation  output 

7. 6. 3. 3  Information  Produced 

•  Federation  output  validation  results 

•  Within  the  context  of  the  Federation  Referent,  identification  of  the  Federation  Acceptability  Criteria 
that  the  derived  federation  output 

•  Meets 

•  Does  not  meet 

•  Characterization  of  the  representational  errors  in  the  derived  federation  output  that  exceed  the  Federation 
Acceptability  Criteria  limits 

•  Estimates  of  the  uncertainties  of  the  representational  errors  in  the  derived  federation  output 

•  Observed  representational  anomalies  and  their  probable  causes 

7.7  Phase  7  -  Consolidate  Federation  W&A  Products 

This  phase  of  the  overlay  focuses  upon  documentation  and  reuse  to  improve  the  productivity  and  efficiency 
of  future  federation  VV&A  processes.  In  support  of  this  goal  the  VV&A  Team  collects  from  the 
preceding  VV&A  efforts  and  assembles  them  into  a  consolidated  package.  This  package  will  include  the  final 
Federation  Accreditation  Plan,  Federation  Referent  description,  final  Federation  Acceptability  Criteria, 
final  Federation  V&V  Plan,  federation  scenario  verification  results,  federation  conceptual  model  V&V  results, 
federation  requirements  verification  results,  federation  design  verification  results,  FOM  verification  results, 
federation  agreement  verification  results,  federate  V&V  results,  federation  infrastructure  verification  results, 
federation  data  set  V&V  results,  integrated  federation  V&V  results,  federation  output  V&V  results, 
and  VV&A  lessons  learned.  This  phase  also  supports  the  data  analysis  efforts  defined  in  the  FEDEP  as 
needed. 

Figure  Al-10  illustrates  the  key  activities  in  this  phase  of  the  VV&A  overlay.  The  subsections  that  follow 
describe  each  of  these  activities.) 
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Figure  A1-10:  Consolidate  Federation  W&A  Products  (Phase  7)  Activity  Diagram. 


7.7.1  Activity  7.1  -  Support  Analyzing  Data 

This  activity  supports  analyzing  the  federation  output.  During  this  activity,  the  VV&A  Team  collects, 
assembles,  and  documents  the  V&V  evidence  from  all  of  the  previous  activities.  They  also  complete 
documenting  the  traceability  between  the  analyzed  data  and  derived  output. 

The  VV&A  documentation  should  include  the  collected  evidence  and  its  relation  to  the  Federation 
Acceptability  Criteria.  An  important  aim  is  to  make  this  documentation  easily  understood  by  User/Sponsor 
and  to  prepare  it  in  an  appropriate  format  as  specified  in  the  V&V  Plan.  Further,  this  documentation  should 
describe  the  conditions  of  recommended  use,  rational  for  those  recommendations,  possible  corrective  actions 
for  follow-on  federation  executions,  and  the  VV&A  lessons  learned. 

7. 7. 1.1  Information  Required 

•  Federation  V&V  results  documentation 

•  Federation  Referent 

•  Federation  Acceptability  Criteria 

7. 7. 1.2  Functions  (T asks) 

•  Collect,  assemble,  and  document  the  verification  and  validation  results  from  prior  activities 

•  Contribute  to  documenting  the  traceability  between  the  analyzed  data  and  the  derived  output 

•  Develop  and  document  the  conditions  of  recommended  use  (i.e.,  accreditation  recommendations  where 
needed) 

•  Develop  recommendations  for  improving  the  VV&A  activities  from  the  lessons  learned 
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7. 7. 1.3  Information  Produced 

•  Integrated  federation  V&V  results  documentation 

•  Information  describing  the  traceability  between  the  analyzed  data  and  the  derived  output 

•  Federation  acceptance/accreditation  recommendations 

•  VV&A  lessons  learned 

•  Recommendations  for  VV&A  activity  improvements 

7.7.2  Activity  7.2  -  Prepare  Federation  VV&A  Products  for  Reuse 

This  activity  prepares  the  federation  products  for  reuse  to  improve  the  productivity  and  efficiency  of  future 
federation  development  and  execution  processes.  The  VV&A  activity  focuses  upon  storing  all  reusable 
federation  products  (including  both  FEDEP  and  VV&A  products)  in  an  appropriate  archive  for  general  reuse 
within  the  domain  or  broader  distributed  simulation  community. 

The  W&A  Team  should  gather  all  of  the  information  that  could  be  re-used  and  store  that  in  an  accessible  form 
in  an  accessible  place.  This  includes  archiving  the  FOM,  any  modifications  to  the  SOMs  of  federates,  and  other 
federation  products  that  may  also  be  reusable,  such  as  the  Federation  Scenarios  and  the  Federation  Conceptual 
Model.  In  fact,  capturing  the  full  set  of  federation  products  required  to  reproduce  the  federation  execution  may 
be  advantageous  in  some  instances.  All  W&A  documents  including  the  Federation  Acceptability  Criteria, 
Accreditation  Plan,  V&V  Plan,  V&V  results  and  acceptance/accreditation  recommendations  should  be  included 
in  the  archive.  These  products  contribute  to  establishing  the  credibility  of  the  federation  as  well  as  the 
federation’s  results.  The  VV&A  Team  should  consolidate  the  products  from  the  federation  VV&A  processes  and 
ensure  that  these  products  conform  to  the  relevant  re-use  exchange  criteria,  formats,  and  available  standards. 
They  should  then  integrate  the  W&A  archive  products  with  those  produced  by  the  Federation  Development 
Team  and  archived  according  to  the  federation  re-use  plan  if  there  is  one. 

The  Federation  Development  Team  should  determine  which  federation  products  have  potential  for  reuse  in 
future  applications  whereas  the  VV&A  Team  should  endeavor  to  capture  all  products  stemming  from  their 
VV&A  activity. 

7. 7.2. 1  Information  Required 

•  Products  of  the  FEDEP  and  VV&A  processes 

•  Guidance  forjudging  the  acceptance  of  products  as  reusable 

7. 7.2.2  Functions  (Tasks) 

•  Consolidate  the  products  produced  by  the  federation  VV&A  processes 

•  Ensure  that  the  VV&A  products  have  been  archived  according  to  the  federation  re-use  plan 

•  Ensure  that  these  products  conform  to  the  relevant  re-use  exchange  criteria,  formats,  and  available 
standards 

•  Confirm  that  the  FEDEP  products  relevant  to  future  VV&A  activities  have  been  archived  according  to  the 
federation  re-use  plan 
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ORGANIZATION 


7. 7.2.3  Information  Produced 

•  Reusable  consolidated  federation  VV&A  products  package 


8.0  CONCLUSION 

This  recommended  practice  has  provided  a  detailed  view  of  the  VV&A  overlay  to  the  FEDEP.  Currently, 
this  model  represents  the  best  practices  available  to  the  VV&A  community.  The  VV&A  overlay  is  a  tailorable 
process  and  is  offered  as  “hands-on”  implementation-level  guidance  to  VV&A  practitioners. 
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